Skip to main content
The control plane keeps everything it knows (apps, domains, tokens, users, encrypted secrets, deploy history) in one SQLite database. A control plane backup is a consistent snapshot of that file, taken while the server keeps running. This is separate from database and volume backups, which protect the data of the apps you deploy. Snapshots here stay on the same disk as the database; see disaster recovery for the off-box, encrypted path that survives losing the machine entirely.

​
What is in a backup, and what is not

Alerting configuration is not in a backup yetalerting.db sits next to levelrail.db in the data directory and holds every alert rule, notification channel (with its webhook URL or key), silence and maintenance window. Neither the local snapshots nor the off-box encrypted backup include it, so a restore on a new machine comes back with apps and users but no alerts. Until that is fixed, copy it yourself on a schedule, for example sqlite3 <data dir>/alerting.db ".backup '/safe/place/alerting.db'" (safe while the control plane runs), and put it back, with the control plane stopped, after a restore. Channel credentials are not in that file: they are envelope encrypted inside levelrail.db, which the backups do include, so restore both files together with the same master key. A channel created after the levelrail.db snapshot you restore comes back with no credential and fails to send; edit it and enter the URL or key again.
The master key is never in a backupSecrets in the database are encrypted with the master key. A restore without the same master key leaves those secrets unreadable. Keep a separate copy of the master key (a password manager or offline vault), and store it apart from the backups. Whoever holds both can read your secrets.
Backups do contain token hashes and encrypted secrets, so every backup route requires a root-scoped token and files are written with mode 0600.

​
Where backups live

Snapshots are written to <data dir>/control-plane-backups/ and named levelrail-YYYYMMDDTHHMMSSZ.db (UTC). Each one is copied with VACUUM INTO, checked with SQLite’s integrity_check, and hashed with SHA-256. Snapshots sit on the same disk as the database. To survive losing the machine, set up encrypted off-box backups, key escrow and restore drills, or download snapshots and store them elsewhere.

​
Automatic snapshots

Manual snapshots (created from the CLI or API) and pre-upgrade snapshots are never deleted automatically.

​
Doctor check

levelrail-cli doctor and the dashboard Status page include a control_plane_backup check. It warns when the newest snapshot is older than 3 days (the fix is levelrail-cli control-plane-backups create), is ok when a recent one exists, and reports unknown when scheduled snapshots are disabled with APP_CONTROL_PLANE_BACKUP_INTERVAL=0. It never fails. See troubleshooting.

​
Stale backup alert

The doctor check only helps when someone looks. To be notified instead, create an alert rule of kind control_plane_backup_stale:
It fires once the newest snapshot is older than 3 days and sends a resolved notice after the next snapshot lands. Set --for-duration (for example 48h) to change the maximum age. The rule is platform-wide (the app only decides where it is listed), stays quiet when APP_CONTROL_PLANE_BACKUP_INTERVAL=0, and also stays quiet before the first snapshot exists. The dashboard’s alert rule dialog and the alerting quick setup prompt offer it too. See observability.

​
Before an upgrade

When the server starts on an existing database and this release carries schema migrations that have not been applied yet, it takes a snapshot first. A brand new database is skipped. If that snapshot fails (for example the disk is full), the failure is logged and the migration still proceeds. To roll back to that snapshot after a bad upgrade, stop the service, install the previous binary and run levelrail restore-snapshot --list, then levelrail restore-snapshot --dry-run latest to verify it, then levelrail restore-snapshot latest (it asks for confirmation; --yes skips the prompt). It uses the same checks and the same .before-restore-<timestamp> safety copy as restore-db. See Installing.

​
Downgrade guard

If the database has a schema version newer than the binary understands, the server refuses to start and says so. Run a newer release, or restore a snapshot taken by this version.

​
Taking and managing backups

Without --out, download writes the raw bytes to stdout. The same operations are available in the dashboard under Settings, Control plane backup, and over the API (see the API reference):

​
Verifying a backup

A backup you have never checked is a hope, not a backup. Verification proves a snapshot is still intact without restoring anything:
  1. checksum: the file’s SHA-256 is recomputed and compared with the checksum recorded when the snapshot was taken. Snapshots from before checksums were recorded pass this check with a note.
  2. integrity: SQLite opens the file read-only and runs integrity_check.
  3. schema_version: the snapshot’s schema version must not be newer than this binary supports, otherwise a restore would be refused.
The command exits 0 when every check passes and 1 when any fails, so it fits in a cron job or a monitoring script. Over the API the response is {"name", "ok", "checks": [{"name", "ok", "detail"}], "verified_at"}; a failed check is still a 200 with "ok": false. The last result is kept in a small file next to the snapshot (no database change) and shows up as verified_at and verified_ok on each entry of the list response. The dashboard shows a Verify button and a last-verified badge per backup. Deleting a snapshot removes its verification record too.

​
Restoring

Restoring is an offline operation on the server, because the database cannot be swapped under a running control plane.
1

Stop the control plane

The database cannot be swapped under a running process.
2

Make the master key available

The restored server must have the same master key, as master.key in the data directory or APP_MASTER_KEY.
3

Run the restore

The command uses APP_DATA_DIR to find the live database.
4

Start the control plane

The reconciler converges your containers toward the restored desired state.
restore-db first checks the file: integrity_check must pass and its schema version must not be newer than the binary. It then moves the current database aside as levelrail.db.before-restore-<timestamp> and puts the backup in place. Nothing is deleted, so a mistaken restore can be undone by moving that copy back. Anything that changed after the snapshot was taken (new apps, tokens, deploys) is gone after a restore. Running containers are not touched, and the reconciler converges them toward the restored desired state on startup.

​
Rolling back to a local snapshot

levelrail restore-snapshot is the friendlier form of the same offline restore for snapshots in the data directory, such as the one taken automatically before a migration:
--list shows snapshots newest first, --dry-run verifies one and changes nothing, and the real run keeps the current database as a .before-restore copy.

​
Restoring on a new machine

A local snapshot is on the dead machine’s disk, so it is of no use there. For a machine that is gone you need the encrypted off-box backup and the escrow bundle: follow the step by step runbook, which was run end to end against a real S3 server. The master key must come with you (escrow bundle or APP_MASTER_KEY); a trailing newline in a recovered master.key is accepted.

​
Next steps

Disaster recovery

Encrypted off-box backups, key escrow, and restore drills.

Master key rotation

Rotate the key that protects stored secrets.
Last modified on October 12, 2026