Skip to main content
Every alert Levelrail sends comes from the control plane itself. If the box loses power, the disk fills, or the process is killed, nothing is left to send “I am down”. The heartbeat closes that gap: while the control plane is healthy it calls a URL you choose every interval. A monitor outside your server alerts when the calls stop.

​
What counts as healthy

A ping goes out only when all of these hold:
  • the database answers,
  • the reconcile loop completed a pass within APP_HEARTBEAT_RECONCILE_WINDOWS (default 3) resync intervals,
  • the local agent is registered (skipped on a fresh install with no local node yet).
If a check fails the ping is held back and the monitor alerts after its grace period. If you also set a failure URL, that URL is called instead, so the monitor can alert at once and name the cause.

​
Setup

Dashboard: Settings, Notification channels, “Off-host heartbeat”. Paste the URL and save. The URL is stored encrypted and only its host is shown afterwards. CLI:
Env (takes precedence over the dashboard, the dashboard then becomes read-only): An invalid URL stops the server at boot with a clear message instead of silently never pinging.

​
Pair it with a monitor

Healthchecks.io: create a check, set its period to your interval and its grace to a few minutes, and use its ping URL (https://hc-ping.com/<uuid>). For the failure URL use the same URL with /fail appended. Uptime Kuma: add a Push monitor, set the heartbeat interval a little above APP_HEARTBEAT_INTERVAL, and use the push URL it shows. Older versions only accept GET on push URLs, so set APP_HEARTBEAT_METHOD=GET. For the failure URL use the same push URL with status=down in the query. Better Stack and similar heartbeat monitors work the same way: copy the heartbeat URL, match the period. Run Uptime Kuma on a different machine or provider than the server it watches. A monitor on the same box dies with it.

​
What is sent

By default a small JSON body: {"status":"ok","version":"...","uptime_seconds":123}. A failure ping adds a one word reason (store, reconcile or agent). No hostnames, no app names, no identifiers. Set APP_HEARTBEAT_BODY=off to send nothing. Redirects to another host are refused, the URL never appears in logs or API responses (only its host), and private destinations are blocked unless you allow them.

​
When only the control plane dies

The heartbeat above is sent by the control plane, so it cannot tell you about a control plane that is dead while the box is still up (for example a crashed process that nothing restarts). Agents cover that: when a node has had no contact with the control plane for APP_AGENT_CP_SILENCE_GRACE, it pings a URL you choose, naming the node in the body, and stops as soon as the session is back. Set it in the dashboard (the “Agent silence URL” field on the same card) or with the CLI:
Use the monitor’s failure endpoint (Healthchecks: the ping URL with /fail appended). The control plane pushes the URL to connected agents over the existing mTLS session, within APP_AGENT_CP_SILENCE_PUSH_EVERY of a change. The agent keeps it in memory only, so an agent restarted while the control plane is already down has no URL until it connects once. The URL is stored encrypted and never logged, only its host. Agent environment: Control plane: APP_AGENT_CP_SILENCE_PUSH_EVERY (default 30s) is how often live sessions re-check the stored URL.

​
What it catches that in-box alerts cannot

Host power loss, kernel panic, OOM kill of the control plane, a full disk that stops the process, a cloud provider outage, a lost network route. It also catches a control plane that is running but wedged, because a stalled reconcile loop stops the pings. It does not tell you which app is down. Keep your normal alerts for that; the heartbeat only answers “is the server itself alive”.
Last modified on October 31, 2026