Hermes "Gateway event loop missed 3 consecutive liveness probes"
Hermes's gateway watchdog exits 75 so systemd or launchd restarts it. Where the stack dump is, how to read what blocked the loop, and what turning it off does.
Hermes Agent’s gateway has a watchdog thread that checks every 30 seconds whether its event loop is still running callbacks. After three missed checks in a row it writes every thread’s stack to ~/.hermes/logs/gateway-shutdown-watchdog.log, marks itself degraded and exits with code 75, which asks systemd or launchd to start it again. The exit is the cure, not the fault: the fault is whatever held the loop, and the first stack in that file names it. You can turn the watchdog off with gateway.loop_watchdog: false, but the stall stays; the gateway just sits frozen instead of restarting.
What the message means
CRITICAL gateway.shutdown_watchdog: Gateway event loop missed 3 consecutive liveness probes; dumping all thread stacks and exiting with code 75 so the service supervisor can restart it.CopiedEverything the gateway does for your chats, its cron scheduler and its kanban dispatcher runs on one asyncio event loop. A separate thread, outside that loop, schedules a tiny callback on it every gateway.loop_watchdog_probe_interval_s seconds (30 by default) and waits up to 10 seconds for it to run. A probe that runs clears the count; one that does not is a strike, and at gateway.loop_watchdog_max_strikes (3 by default) the watchdog acts. That is the behaviour in Hermes’s messaging docs and in the watchdog’s code at v0.23.3. So by the time the line is logged, nothing on the loop has run for well over a minute: no replies, no scheduled jobs.
Code 75 is EX_TEMPFAIL, which Hermes uses to “ask the service manager to restart”. The generated systemd unit restarts it and launchd’s KeepAlive relaunches it. On Windows the docs say the scheduled task’s restart policy only covers a gateway that fails to start, so a gateway that exits later stays down until you run hermes gateway start.
Until the new process overwrites it, hermes gateway status shows “Gateway exited degraded: event loop stopped dispatching”, the dashboard badge reads Degraded, and gateway_state.json carries exit_reason: loop_liveness_watchdog. One such exit followed by a working gateway is the watchdog doing its job. Repeated ones mean something keeps blocking the loop.
Find the call that held the loop
The dump goes to gateway-shutdown-watchdog.log in the logs folder of your Hermes home (~/.hermes/logs/ unless you set HERMES_HOME), and a copy goes to the gateway’s stderr, which a systemd service sends to the journal:
less ~/.hermes/logs/gateway-shutdown-watchdog.logCopiedIn v0.23.3 the file opens with a section headed --- monitored gateway loop thread stack ---, then every thread. That first stack is the one that matters. Find its innermost call, the one the loop thread was inside when it stopped: a network connect, a file read, a database query, a subprocess or a lock. The file and function names in it say which part of Hermes was waiting: a messaging adapter, the credential store, a plugin. If the loop thread looks idle but other threads are busy importing, globbing or compressing, the loop was probably waiting for Python’s interpreter lock rather than blocked by its own code.
If the gateway is frozen right now and you want a look before it is killed, kill -USR2 <gateway pid> appends every thread’s stack to ~/.hermes/logs/gateway_faulthandler.log and leaves the gateway running (Linux and macOS).
What people have found blocking it
These are users’ reports on Hermes’s tracker from November 2026, covering v0.22.x to v0.23.3. All are open and none has a cause confirmed by a maintainer, so treat them as places to look, not diagnoses.
- Telegram’s connection check. On v0.23.1, one dump showed the loop thread inside the Telegram adapter’s heartbeat,
bot.get_me(), stuck in a TCP connect while Telegram requests were timing out. Its 15-second timeout runs on the same loop, so it could not fire. The log before it shows Telegram timeouts. - Worker threads starving the loop. On macOS, reports describe the loop thread spending most of its time waiting for the interpreter lock while other threads ran a YAML import that globs site-packages and compressed session state (#157764, v0.23.2), or with 15 profiles served by one gateway (#153727). A replacement that takes minutes to boot can then stall the same way and loop.
- File and database work on the loop. On Windows with antivirus scanning every file, path checks,
.envreads and SQLite writes on the loop thread (#153063); with many concurrent sessions on macOS, the credential pool reading.envand the Keychain on every Anthropic request (#157699); and a slow kanban query on a large board, which one user cleared with an index ontask_runs. - Secret managers. With profiles pulling keys from 1Password, one profile’s slow
op readheld a lock that every incoming message waited on (#152902). - Memory. A busy Linux server with 30 to 40 clients saw the gateway grow from about 500 MB to over 8 GB before the watchdog restarts (#149903), and several reports name swap or a large state database as making stalls worse.
If your stack matches none of these, the issue with the closest frames is the place to add yours, with the dump attached.
Turning the watchdog off
The documented switch is one line under gateway in config.yaml:
gateway:
loop_watchdog: falseCopiedIt does not remove the stall. With the watchdog off, a blocked loop stays blocked, and a gateway that looks alive answers nothing until you restart it; the only sign left is hermes gateway status warning that the heartbeat is stale once housekeeping has not run for 120 seconds. On Linux the docs offer a systemd-side alternative: set gateway.systemd_watchdog_seconds (for example 120), then run hermes gateway install --force to regenerate the unit, and systemd restarts the gateway when the loop stops sending it heartbeats.
When the restart itself is slow
Exit 75 only helps if the process exits and the supervisor brings it back. One report found the exit can hang while writing the dump to a full stderr pipe, and on macOS another saw the dump take about five minutes to appear. If a gateway shows no log lines and no exit for minutes, take a SIGUSR2 dump, then stop it with hermes gateway stop and start it again. On a Linux user service, check that it is set to survive your logout (lingering), and if the restarted gateway refuses because another still holds the host, that is the gateway lock, which clears once the old process is gone.