Skip to main content
Levelrail keeps your secrets encrypted, signs every request in with a session or a scoped token, and never reaches into your servers over SSH after setup. This page explains how those pieces fit together and links to the detailed page for each. It is a map, not a duplicate.

​
Five things to do on a new install

  1. Serve the dashboard over HTTPS and set the dashboard URL. Once an https:// URL is set, sign-in over plain HTTP is refused. See Domains and ingress.
  2. Give people the narrowest ability that works. Most humans need write and deploy; keep root for a few. Give CI and AI tokens one ability, and attach a Deny for production apps. See Identity and access.
  3. Turn on two-factor sign-in for every account that holds root.
  4. Pin the control plane on every agent by setting APP_CA_FINGERPRINT, so enrollment cannot be intercepted. See Multi-node.
  5. Back up the master key separately from the data directory, and keep it out of the same snapshot. See Master key rotation and Control plane backup.
The Security center in the dashboard checks most of these from your instance’s real state. For what Levelrail defends against and where it does not yet, read the threat model. Fixed issues are listed on the security advisories page.

​
Secrets: envelope encryption

Every owner of secrets (an app, a backup target, the email settings, and so on) gets its own random data encryption key (DEK), and each of its values (an app env var marked secret: true, email credentials, API tokens) is encrypted under that DEK with AES-256-GCM. Every DEK is wrapped under one master key. The control plane loads that key into memory at startup. By default the key is generated on first start and stored in <data dir>/master.key (mode 0600); set APP_MASTER_KEY to supply it from your own secret manager instead. It is never included in a control plane backup snapshot.
Anyone who can read your whole data directory (a disk image, a host-level backup, a cloud snapshot) can read master.key next to the database and decrypt every secret. Keep that key file out of host backups and snapshots, or store it elsewhere with APP_MASTER_KEY_FILE=/etc/levelrail/master.key (an absolute path outside the data directory), or supply it with APP_MASTER_KEY from a secret store.
Each encrypted value is also bound to the slot it was written to: its owner and key name are sealed inside the ciphertext and checked on every read. Someone with write access to the database cannot copy one app’s DATABASE_URL ciphertext into another app’s API_KEY row and have it decrypt there; the read fails closed instead. Values written before this existed are “legacy” and should be bound once with levelrail-cli secrets rebind, see Binding secrets to their slot.
Rotating the master key re-wraps every DEK without ever exposing plaintext, then binds any legacy values. See Master key rotation for the full procedure and failure modes.
Sensitive plain variables and secrets are masked on every read surface, and reading one back needs a confirmed, audited step-up. See Environment variables and secrets. Credentials for backup targets and registry integrations follow the same write-only pattern described in Backups and storage: once saved, the plaintext is never returned by the API again, only a masked placeholder.

​
Notification channel secrets

A notification channel’s credential (its webhook URL, Telegram bot token, PagerDuty routing key, Opsgenie or Resend key, and a Slack signing secret or Discord public key for interactive approvals) is envelope encrypted like any other secret, under the channel’s own slot. alerting.db keeps only a masked placeholder such as https://hooks.slack.com/(hidden), and the API returns that placeholder to every caller, admin included. The value is decrypted only at the moment a notification, test message or approval request is sent, or an interactive button click is verified. An email channel’s recipient address is not a credential and stays readable. Channels saved by an older release are encrypted in place on the first start with a master key: each value is written as ciphertext, decrypted again to verify it, and only then is the plaintext column cleared. The step is idempotent and safe to interrupt; anything it could not finish is retried on the next start. Without a master key, channels keep working with plaintext credentials and levelrail-cli doctor warns with a notification_channel_secrets check until one is configured.

​
Sessions and tokens

  • Session cookies are HttpOnly, SameSite=Lax, and Secure whenever the request arrived over HTTPS (directly or through the embedded Caddy ingress). Once an https:// dashboard URL is set, sign-in over plain HTTP is refused (APP_ALLOW_INSECURE_LOGIN=true is the recovery escape hatch).
  • Cross-origin guard: a state-changing request (POST, PUT, PATCH, DELETE) that authenticates with the session cookie and carries an Origin header naming a different host than the one served is rejected with 403. Bearer-token and CLI requests, which send no cookie or no Origin, are unaffected.
  • First admin registration requires the one-time setup token from <data dir>/setup-token, so an exposed fresh install can’t be claimed by a stranger.
  • Rate limits use the real client address. Behind the embedded ingress the control plane reads the right-most untrusted hop of X-Forwarded-For, but only when the TCP peer is loopback or listed in APP_INGRESS_TRUSTED_PROXIES. Any other peer cannot spoof its address, so one visitor can no longer exhaust the login or webhook budget for everyone.
  • Token-redeeming routes are rate limited. POST /api/v1/auth/reset-password and POST /api/v1/invites/accept are unauthenticated by design, so each gets a per-client-IP budget (APP_API_RATE_LIMIT_TOKEN_REDEEM_RPM, default 10 per minute, 0 disables). Over budget returns 429 with Retry-After.
  • API tokens are minted per-user, scoped by ability (a user can only mint a token holding abilities they hold themselves), and can be issued through a device-code flow for headless environments.
  • Two-factor authentication (TOTP) is available per user, with recovery codes for account lockout.
  • Two-factor codes are single-use. A TOTP code that was already accepted is refused if presented again inside its validity window, and recovery codes are consumed on use.
  • Read-tier callers cannot enumerate people or credentials. GET /api/v1/users returns only the caller’s own record unless the caller is root, and notification destinations (webhook URLs, bot tokens, routing keys) are shown as (hidden) without read:sensitive, see observability.
Full detail on all three: Identity and access.

​
Authorization: abilities, roles, and IAM policies

Levelrail layers two permission models:
  • Abilities: a flat list (AbilityRead, AbilityWrite, AbilityDeploy, AbilityRoot, and so on) attached to a user or token. Cheap to check, coarse-grained.
  • IAM policies: AWS-IAM-shaped Allow/Deny documents scoped to a specific app or database, for when a flat ability isn’t precise enough.
Evaluation order, the full ability list, and policy examples: Identity and access.

​
TLS and network exposure

  • Certificates are issued and renewed automatically through embedded Caddy’s ACME client. See the ACME verification runbook if issuance fails.
  • HSTS (Strict-Transport-Security) is opt-in, not on by default, because turning it on for a domain that later loses TLS locks users out until the header expires. See Domains and ingress.
  • WAF mode and rate limiting are configurable per domain, with a detect-only mode for testing rules before enforcing them. See Domains and ingress.
  • The node agent dials out to the control plane. No inbound ports need to be open on a managed server for enrollment or day-to-day operation.
  • Agent enrollment pins the control plane CA. A join token is shown together with the agent CA’s SHA-256 fingerprint; with APP_CA_FINGERPRINT set, the agent checks the control plane’s certificate against that CA before it sends the token, so an attacker in the network path cannot capture the token or pose as the control plane. Without the fingerprint the agent falls back to trust on first use and logs a warning. After enrollment every connection is mutual TLS against the saved CA.
  • Container installs bind plain HTTP to loopback. The committed docker-compose.yml publishes 8080 on 127.0.0.1 only (override with LEVELRAIL_HTTP_BIND), see Docker.
  • install.sh fails closed on checksums. A release without a usable checksums.txt aborts the install unless you pass LEVELRAIL_SKIP_CHECKSUM=1. Release checksums are cosign-signed; APP_INSTALL_VERIFY=require makes the installer insist on a valid signature, see Verifying release binaries.

​
Outbound requests to user-supplied URLs

Alert rules, deploy notifications, and notification channels post to URLs an operator types in. To keep those from being pointed at the control plane’s own network (cloud metadata at 169.254.169.254, the Docker API, databases on a private subnet), every notification request refuses to connect to loopback, private (RFC 1918, fc00::/7), link-local, CGNAT, multicast, and other reserved addresses. The check runs on the IP actually dialed, after DNS resolution and on every redirect hop, so a hostname that resolves (or later re-resolves) to an internal address is caught too. Proxy environment variables are ignored for these requests for the same reason. If you run a notification receiver on your own network (a self-hosted Mattermost, Gotify, or ntfy on a private IP, for example), set:
on the control plane. This re-allows every internal address for notification requests, so only enable it when every user who can create alert rules or notification channels is trusted with access to that network. The same check runs when you create or edit a notification channel: a URL that is not http(s), or that is a loopback, private or link-local address (or localhost, or a hostname that resolves only to those), is refused up front with a 400 that names APP_NOTIFY_ALLOW_PRIVATE_NETWORKS, instead of being saved and failing silently on every send. Set the variable before creating the channel when the receiver really is on a private network. Email notifications are separate: recipient addresses must be a single plain address, and subjects are MIME-encoded, so user input cannot add mail headers.

​
Fresh-box hardening checklist

A short list for the server itself, independent of anything Levelrail configures:
  • SSH key auth only. Disable password login (PasswordAuthentication no in sshd_config) before exposing the box to the internet. Neither install.sh nor the agent touch SSH configuration; this is standard server hygiene, not something Levelrail does for you.
  • Only the control-plane node needs inbound ports. 80/tcp and 443/tcp for ingress (the ACME HTTP-01 challenge plus HTTPS traffic), your SSH port, and nothing else. See Installing: requirements for the full list and install.sh’s optional ufw setup.
  • levelrail-cli doctor’s firewall check detects ufw, firewalld, and nftables/iptables, whichever is actually installed on this host, and reports a concrete command to open 80/443 when it isn’t. No supported tool installed is reported as informational, not a failure: this platform can’t tell whether you’re relying on something it can’t introspect (a cloud security group, for example).
  • Agent-only nodes should accept no inbound traffic at all. The node agent dials out to the control plane over mTLS; the control plane never initiates a connection. A server running only the agent has nothing that needs to accept a connection, so leave its firewall closed by default rather than opening anything speculatively.
  • Don’t expose anything Levelrail didn’t ask you to. Docker’s daemon socket, the SQLite database file, and the mesh DNS listener (:5390 by default, see Multi-node) are all meant to stay local to the box.
  • Keep the OS patched. The patch_status alert (see Observability) can warn you about pending security patches on a managed node, but it only reports, it doesn’t apply anything; that’s still on you (unattended-upgrades, dnf-automatic, or your distro’s equivalent).

​
Audit log

Every mutating API call from an authenticated principal is recorded: who, what, when, and the outcome. The log is queryable and exportable as CSV. See Identity and access.

​
Container hardening

Every container Levelrail creates (apps, databases, catalogue services, helpers) can be created with secure defaults: all Linux capabilities dropped except a minimal set, no-new-privileges, and a per-container process limit. It is controlled by APP_CONTAINER_HARDENING on the control plane and on each agent: enforce is the default: an image that needs a capability outside the minimal set fails when it starts, and the fix is APP_CONTAINER_HARDENING_CAP_ADD (or APP_CONTAINER_HARDENING=warn while you investigate). The stock nginx, postgres, mysql, mongo and redis images were verified to start under these settings. What enforce sets:
  • CapDrop: ALL, then CapAdd: CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETGID, SETUID. This is what the stock nginx, postgres, mysql, redis, node and python images need for their entrypoints (chown a data dir, drop privileges to a service user, bind port 80 or 443). Docker’s default set also grants NET_RAW (raw sockets, ping), MKNOD, SETFCAP, SETPCAP, SYS_CHROOT and AUDIT_WRITE, which are left out.
  • Capabilities a caller already asks for (the egress sidecar’s NET_ADMIN) are kept.
  • SecurityOpt: no-new-privileges, so a process cannot gain privileges through setuid binaries.
  • PidsLimit from APP_CONTAINER_PIDS_LIMIT (default 4096, 0 disables), which stops a fork bomb from exhausting the host.
APP_CONTAINER_HARDENING_CAP_ADD (comma separated, for example SYS_CHROOT,NET_RAW) adds capabilities for every container on that host, for images that need more than the minimal set (SSH or FTP servers, tools using ping). It applies host-wide because a per-app override would need a new field carried to remote agents; a per-app security block in app.yaml is future work. Read-only root filesystems and privileged containers are never set by these defaults. The setting is read by the process that creates the container, so it applies to remote nodes when set on the agent, and the doctor reports the control plane’s own value.

​
Docker API guard

The control plane and each agent reach Docker through an in-process allowlisting proxy that refuses privileged containers, host namespaces, unsafe capabilities and protected host mounts. It ships in enforce mode, with APP_DOCKER_GUARD=audit and off as overrides. See Docker access and the API guard for the endpoint list, every rule, and its limits.

​
Rootless and Podman

GET /api/v1/system/doctor’s container_runtime check (levelrail-cli doctor) reports which container engine and privilege mode the control plane is talking to: Docker or Podman, rootful or rootless, and which setting decided the socket (APP_CONTAINER_RUNTIME_SOCKET, then the standard DOCKER_HOST, then a well-known rootless or Podman socket path, then the rootful default /var/run/docker.sock). Podman exposes a Docker-compatible Engine API socket, so the existing Docker Engine API client talks to it unchanged, no CLI shelling and no new client library, consistent with the rule that Levelrail never shells out to the Docker CLI. APP_CONTAINER_RUNTIME_SOCKET pins the connection to a specific socket (for example unix:///run/user/1000/podman/podman.sock) ahead of DOCKER_HOST, for operators who would rather not export DOCKER_HOST into the whole process environment. Container hardening (above) adjusts automatically under detected rootless: PidsLimit is disabled by default, because rootless Docker without a delegated cgroup v2 controller rejects a pids-limit HostConfig at container-create time. Set APP_CONTAINER_PIDS_LIMIT explicitly to override this. CapDrop, CapAdd and no-new-privileges are unaffected: those apply inside the container’s own user namespace regardless of rootless mode, so nothing needed to come off the minimal capability list. What is not verified: this detection and the hardening adjustment above were tested against a fake Docker client, not a real rootless Docker or Podman installation. Known gaps to check before relying on this in production:
  • Bind-mount ownership. A bind mount’s files keep the UID/GID they have on the host; under rootless Docker/Podman that UID sits inside a remapped user namespace, so a container process expecting to own a bind-mounted path can see a permission mismatch that doesn’t happen under rootful Docker. Named volumes (Levelrail’s default for managed databases) aren’t affected the same way.
  • Low host ports. Rootless port publishing goes through a userspace proxy (rootlesskit/slirp4netns), which by default can’t bind host ports below 1024 regardless of the container’s own capabilities. The embedded ingress runs on the host directly, not in a rootless container, so this only affects an app that needs to publish a low host port itself.
  • cgroup limits beyond PidsLimit. Memory and CPU limits also need a delegated cgroup v2 controller under rootless; unlike PidsLimit, this isn’t currently detected or adjusted for Resources.MemoryBytes/NanoCPUs.
Treat single-container rootless or Podman hosts as best-effort, not fully verified, until tested against a real installation.

​
Other hardening details

A cookie-authenticated state-changing request is refused when its Origin names another host, when Origin is null, when there is no Origin but Sec-Fetch-Site says cross-site or same-site, or when there is no Origin and the Referer names another host. Requests with a bearer token or no session cookie are not subject to the check.
Redirect targets and custom error pages are written to the ingress with { and } escaped, so a placeholder such as {env.NAME} or {file.path} in operator text can never be expanded into a control plane secret.
Stored certificates from other CAs are purged only when the configured CA changes, not on every start, so a restart never deletes valid certificates or spends Let’s Encrypt rate limit. The sslip.io hostname is only offered for a publicly routable address (carrier-grade NAT, benchmarking and reserved ranges are refused).
In addition to /etc, /root and the Docker socket, /run, /dev, the directories that hold executables (/bin, /sbin, /lib, /usr/bin, /usr/local/bin and similar), /var/spool/cron, the control plane and agent data directories (including a custom APP_DATA_DIR), and any parent directory that would contain a protected path (/var, /var/lib) are refused. Paths are checked lexically: a symlink on the host that points into a protected path is not followed, so do not create such symlinks inside a directory you let apps mount.
The built-in registry’s plain HTTP port is published on loopback only. Remote nodes pull through the TLS route. An existing container published on all interfaces is recreated.
repo_url clones, branch listing and pipeline checkouts go through the same guarded HTTP client as webhooks: internal, loopback, link-local and metadata addresses are refused, including after redirects and DNS changes. A self-hosted Git server on a private address needs APP_GIT_ALLOW_PRIVATE_NETWORKS=true, which relaxes only git fetches. The older APP_NOTIFY_ALLOW_PRIVATE_NETWORKS=true still works for git but also relaxes notification webhooks, so prefer the git-specific one.
Private keys, bearer tokens, token-shaped strings and credentials in URLs are masked before a build or clone log line is stored or streamed.
A node name must be 1 to 63 letters, digits, dot, underscore or hyphen, and control_plane_addr for provisioning must be host:port, so neither can inject lines into the agent’s environment file.
Data and install directories must be absolute, free of shell and unit metacharacters, and the data directory may not be a system directory (it is the target of rm -rf on uninstall --purge). Ports must be numbers in range and LEVELRAIL_VERSION is restricted to a safe character set.

​
Reporting a vulnerability

Levelrail does not yet have a dedicated security disclosure address. Until one exists, open a private security advisory on the GitHub repository rather than a public issue. The repository’s SECURITY.md has the full policy, including what to expect after reporting.

​
What this does not cover yet

  • No SSO/SAML or SCIM. Sign-in is local password, passkey, or OAuth (Google, GitHub, Microsoft, generic OIDC).
  • No secret scanning of an app’s own source repository.
  • Sessions live in memory, so every control plane restart (including an upgrade) signs everyone out. API tokens are unaffected.
  • GET /api/v1/system/containers lists containers Levelrail does not manage only to root callers. Other callers with read see managed containers only.
  • Container hardening is on by default but has no per-app override yet, and read-only root filesystems stay opt-in.
  • Rootless Docker and Podman detection and hardening adjustment (above) exist but are not verified against a real installation; bind-mount ownership and cgroup resource limits beyond PidsLimit are known gaps.

​
Next steps

Identity and access

Users, tokens, roles, and IAM policies.

Threat model

Trust boundaries, mitigations with file references, and known gaps.

Master key rotation

Rotate the key that protects all secrets.

Domains and ingress

TLS certificates and domain configuration.
Last modified on October 12, 2026