GuidesHosting choices

Hermes "Gateway event loop missed 3 consecutive liveness probes"

Hermes's gateway watchdog exits 75 so systemd or launchd restarts it. Where the stack dump is, how to read what blocked the loop, and what turning it off does.

November 20, 2026The Everpod team
The short answer

Hermes Agent’s gateway has a watchdog thread that checks every 30 seconds whether its event loop is still running callbacks. After three missed checks in a row it writes every thread’s stack to ~/.hermes/logs/gateway-shutdown-watchdog.log, marks itself degraded and exits with code 75, which asks systemd or launchd to start it again. The exit is the cure, not the fault: the fault is whatever held the loop, and the first stack in that file names it. You can turn the watchdog off with gateway.loop_watchdog: false, but the stall stays; the gateway just sits frozen instead of restarting.

What the message means

CRITICAL gateway.shutdown_watchdog: Gateway event loop missed 3 consecutive liveness probes; dumping all thread stacks and exiting with code 75 so the service supervisor can restart it.
Copied

Everything the gateway does for your chats, its cron scheduler and its kanban dispatcher runs on one asyncio event loop. A separate thread, outside that loop, schedules a tiny callback on it every gateway.loop_watchdog_probe_interval_s seconds (30 by default) and waits up to 10 seconds for it to run. A probe that runs clears the count; one that does not is a strike, and at gateway.loop_watchdog_max_strikes (3 by default) the watchdog acts. That is the behaviour in Hermes’s messaging docs and in the watchdog’s code at v0.23.3. So by the time the line is logged, nothing on the loop has run for well over a minute: no replies, no scheduled jobs.

Code 75 is EX_TEMPFAIL, which Hermes uses to “ask the service manager to restart”. The generated systemd unit restarts it and launchd’s KeepAlive relaunches it. On Windows the docs say the scheduled task’s restart policy only covers a gateway that fails to start, so a gateway that exits later stays down until you run hermes gateway start.

Until the new process overwrites it, hermes gateway status shows “Gateway exited degraded: event loop stopped dispatching”, the dashboard badge reads Degraded, and gateway_state.json carries exit_reason: loop_liveness_watchdog. One such exit followed by a working gateway is the watchdog doing its job. Repeated ones mean something keeps blocking the loop.

Find the call that held the loop

The dump goes to gateway-shutdown-watchdog.log in the logs folder of your Hermes home (~/.hermes/logs/ unless you set HERMES_HOME), and a copy goes to the gateway’s stderr, which a systemd service sends to the journal:

less ~/.hermes/logs/gateway-shutdown-watchdog.log
Copied

In v0.23.3 the file opens with a section headed --- monitored gateway loop thread stack ---, then every thread. That first stack is the one that matters. Find its innermost call, the one the loop thread was inside when it stopped: a network connect, a file read, a database query, a subprocess or a lock. The file and function names in it say which part of Hermes was waiting: a messaging adapter, the credential store, a plugin. If the loop thread looks idle but other threads are busy importing, globbing or compressing, the loop was probably waiting for Python’s interpreter lock rather than blocked by its own code.

If the gateway is frozen right now and you want a look before it is killed, kill -USR2 <gateway pid> appends every thread’s stack to ~/.hermes/logs/gateway_faulthandler.log and leaves the gateway running (Linux and macOS).

What people have found blocking it

These are users’ reports on Hermes’s tracker from November 2026, covering v0.22.x to v0.23.3. All are open and none has a cause confirmed by a maintainer, so treat them as places to look, not diagnoses.

If your stack matches none of these, the issue with the closest frames is the place to add yours, with the dump attached.

Turning the watchdog off

The documented switch is one line under gateway in config.yaml:

gateway:
  loop_watchdog: false
Copied

It does not remove the stall. With the watchdog off, a blocked loop stays blocked, and a gateway that looks alive answers nothing until you restart it; the only sign left is hermes gateway status warning that the heartbeat is stale once housekeeping has not run for 120 seconds. On Linux the docs offer a systemd-side alternative: set gateway.systemd_watchdog_seconds (for example 120), then run hermes gateway install --force to regenerate the unit, and systemd restarts the gateway when the loop stops sending it heartbeats.

When the restart itself is slow

Exit 75 only helps if the process exits and the supervisor brings it back. One report found the exit can hang while writing the dump to a full stderr pipe, and on macOS another saw the dump take about five minutes to appear. If a gateway shows no log lines and no exit for minutes, take a SIGUSR2 dump, then stop it with hermes gateway stop and start it again. On a Linux user service, check that it is set to survive your logout (lingering), and if the restarted gateway refuses because another still holds the host, that is the gateway lock, which clears once the old process is gone.

Hermes installed on an always-on developer pod, which you run yourself.

A developer pod is a permanent cloud computer (VPS), set up and secured for you, with your pick of Claude Code, Codex, OpenCode, Pi, Hermes and OpenClaw installed. Run more agents at once than your laptop can hold, and keep them working when it’s closed. From $24 a month, built in about ten minutes.