# About Levelrail: open source, built by GLINCKER
Source: https://docs.levelrail.com/about
Levelrail is an Apache 2.0 licensed, self-hosted deployment platform built by GLINCKER. Read what it is, how it is licensed, and how pre-release it still is.
**Levelrail is built by GLINCKER, in the open**
A self-hosted deployment platform for your own Linux servers. It is Apache 2.0 licensed and developed in public on GitHub.
[View on GitHub](https://github.com/glincker/levelrail) | [Roadmap](/roadmap)
## Who builds it
Levelrail is a GLINCKER project. The source, issues and discussions live at github.com/glincker/levelrail.
## License
Apache 2.0, one edition, no paywalled features. See Pricing for what that means in practice.
## Honest status
Levelrail is pre-release software. The roadmap and feature status pages say what is shipped, what is experimental and what is not proven yet, and the changelog lists every release.
## Related
* [Roadmap](/roadmap)
* [Changelog](/changelog/)
* [Feature status](/feature-status)
* [Pricing](/pricing)
* [Contact](/contact)
* [Contributing](https://github.com/glincker/levelrail/blob/main/CONTRIBUTING.md)
## Help shape it
Read the contributing guide, open an issue, or try it on a spare server.
[View on GitHub](https://github.com/glincker/levelrail) | [Roadmap](/roadmap)
# Access control
Source: https://docs.levelrail.com/access-control
Roles for multi-user access, per-user environment grants, the guest role, IAM policies on apps, databases and environments, and ready-made policy templates.
Give each teammate the access they need and no more. A role bundles abilities, environment grants narrow a guest down to chosen environments, and IAM policies allow or deny access to individual apps, databases and environments.
Managing stored roles and environment grants is behind the `access-roles` experimental flag: set `APP_EXPERIMENTAL=access-roles` on the control plane (see [Experimental features](/experimental-features)). Listing roles, IAM policies and the environment rules below are always on.
## Built-in roles
Four roles exist on every instance. They cannot be edited or deleted.
| Role | Abilities | Visibility | Use for |
| - | - | - | - |
| `admin` | `root` | all | Full control |
| `operator` | `read`, `read:sensitive`, `write`, `deploy` | all | Deploy and configure |
| `viewer` | `read` | all | Read-only access |
| `guest` | `read` | granted | Read-only access to the environments you grant |
## Abilities
Every user and every API token carries abilities. A role is a named set of them.
| Ability | Allows |
| - | - |
| `read` | View apps, logs, metrics, deploy history |
| `read:sensitive` | View secrets and other private config |
| `write` | Change config, restart, manage domains and env vars |
| `write:sensitive` | Rotate secrets and update sensitive values |
| `deploy` | Trigger a deploy |
| `root` | Everything. Exclusive: it cannot be combined with other abilities |
| `signin:approve` | Tokens only: show your own sign-in codes and approve new browsers. Root does not include it, and only a signed-in session can mint it |
| `secrets:reveal` | Tokens only: reveal secret values together with `read:sensitive`, the owner's password at request time, and `APP_SECRET_REVEAL_TOKENS=on`. Root does not include it, and only a signed-in session holding `read:sensitive` can mint it. See [Environment variables and secrets](/environment-and-secrets#api-tokens) |
See [Identity and access](/identity-and-access) for how abilities, sessions and tokens work.
## Custom roles
Create a role with a name, the abilities it holds and a visibility:
```bash theme={"dark"}
levelrail-cli roles create "Platform lead" \
--abilities read,read:sensitive,write,deploy \
--description "Deploys and configures, never rotates secrets"
levelrail-cli roles list
levelrail-cli roles update "Platform lead" --abilities read,write,deploy
levelrail-cli roles delete "Platform lead"
```
A role can be given by name or id. In the dashboard, open Settings, Team, Roles.
Rules the server enforces:
* Built-in roles cannot be edited or deleted.
* A role that is assigned to users cannot be deleted. Reassign them first.
* Editing a custom role updates the abilities of everyone who holds it, in one step.
* The last user with `root` cannot lose it, whether by a role change, a role edit or a deletion.
* You cannot change your own role. Ask another admin.
## Assign a role to a user
```bash theme={"dark"}
levelrail-cli users create --email ops@example.com --password 'change-me-now' --role operator
levelrail-cli users role set user_abc operator
```
Invites accept a role too. Editing a user's abilities by hand removes their role label, because the abilities no longer match any role.
## Visibility and the guest role
A role's visibility is `all` (the default) or `granted`.
* With `all`, a user sees every app and database their abilities allow.
* With `granted`, a user sees only apps and databases in the environments granted to that user. A role with `granted` visibility may hold only the `read` ability.
Anything a guest cannot see answers 404, not 403, so they cannot even tell it exists. Resources with no environment are hidden from guests. API tokens follow the visibility of the user who owns them. The apps, databases, deployments and approvals lists are filtered the same way.
### Environment grants
```bash theme={"dark"}
levelrail-cli users grants set user_abc --environment env_dev --environment env_test
levelrail-cli users grants get user_abc
levelrail-cli users grants set user_abc # no --environment: clear all grants
```
`grants set` replaces the whole list. A user with no grants sees nothing. Grants only apply to users whose role has `granted` visibility. In the dashboard, the Users page shows an environment checklist when you change a user to such a role. See [Environments](/environments) for the environments you can grant.
## IAM policies
A policy adds Allow or Deny rules on top of a user's or token's abilities:
```json theme={"dark"}
{
"Statement": [
{ "Effect": "Allow", "Action": ["deploy"], "Resource": ["app:checkout"] },
{ "Effect": "Deny", "Action": ["deploy"], "Resource": ["environment-kind:production"] }
]
}
```
Evaluation order: an explicit Deny wins, then the user's own abilities, then a matching Allow.
Resource strings:
| Resource | Matches |
| - | - |
| `app:{name}` | One app |
| `database:{name}` | One database |
| `environment:{id}` | Every app and database tagged with that environment |
| `environment-kind:{kind}` | Every app and database in any environment of that kind: `dev`, `test`, `uat`, `production`, `preview` or `custom` |
A trailing `*` matches a prefix (`app:*`, `environment:*`, `environment-kind:*`). An app or database with no environment matches no environment rule.
```bash theme={"dark"}
levelrail-cli iam policies create --name no-prod-deploys \
--document '{"Statement":[{"Effect":"Deny","Action":["deploy"],"Resource":["environment-kind:production"]}]}'
levelrail-cli iam policies attach pol_abc --principal-type user --principal-id user_xyz
levelrail-cli iam policies detach pol_abc --principal-type user --principal-id user_xyz
```
### Policy templates
Five ready-made policies. In the dashboard use Settings, IAM policies, Add from template: it shows the effect in plain language and the JSON before you create it.
| Template | Effect |
| - | - |
| `read-only` | Read access only |
| `guest-one-environment` | Read access to one environment (parameter `environment`, which must exist) |
| `deployer-nonprod` | Read, write and deploy on `dev`, `test`, `uat` and `preview`. Denies write, `write:sensitive` and deploy on `production` |
| `ai-operator-nonprod` | Same allow as `deployer-nonprod`. Denies every mutating ability on `production` and `root` everywhere |
| `production-approver` | Read and deploy on `production` |
```bash theme={"dark"}
levelrail-cli iam templates list
levelrail-cli iam templates apply guest-one-environment --param environment=env_dev --attach-user user_abc
```
`apply` accepts `--name`, `--attach-user` and `--attach-token`. A duplicate policy name answers 409.
## Audit
Every request above `read` is written to the audit log with who made it and which ability it needed:
```bash theme={"dark"}
levelrail-cli audit-log --client-kind cli --method POST
levelrail-cli audit-log --format csv --output-file audit-export.csv
```
## Build, test and review policies
Open **Settings, IAM policies** for the policy workspace. How the evaluator works is written down in [IAM design notes](/iam-design).
* Guided builder: pick Allow or Deny, choose abilities from a grouped list with a plain description and a risk hint, then choose resources by picker: everything, a project (expanded to its environments), an environment, an environment kind, one app, one database or a name pattern. Each choice shows how many apps and databases it matches today. The stored JSON is shown beside the rules and is only editable behind the **Edit JSON** switch. Errors point at the field path.
* Templates with parameters: read only, guest in one environment, deployer for non-production, operator of one app, owner of one database and deployer for one project. You pick the app, database, environment or project instead of typing an ID.
* Simulator: ask whether a user or token can do an ability on a resource. The answer comes from the same evaluator that guards requests and lists the deciding statement and every statement that matched. The same check runs from the CLI and the read-only MCP tools.
* Who has access: start from a user or token: its abilities, attached policies and what it can do on every app and database. Attach or detach with a preview of the access that changes, in bulk for tokens.
* Analyzer: static checks with a severity and a one line fix: allow everything, wildcard or root abilities, sensitive abilities on every resource, a Deny that can never match, unattached policies, tokens holding root, attachments to a missing user or token, shadowed or overlapping statements and keys the evaluator ignores.
* Safe changes: every saved document is kept as a version with a rule diff. A change that would leave no operational root principal is refused.
Conditions such as source IP, time window and MFA are not part of the evaluator, so the builder does not offer them. Scope by environment with resources instead, and use a [session policy](session-policy.md) for allowed networks and required MFA.
```bash theme={"dark"}
levelrail-cli iam simulate --principal token:TOKEN_ID --action deploy --resource app:web
levelrail-cli iam effective --principal user:USER_ID
levelrail-cli iam analyze --json
```
## API
| Method | Path | Ability |
| - | - | - |
| `GET` | `/api/v1/roles` | `read` |
| `POST` | `/api/v1/roles` | `root` |
| `PUT` | `/api/v1/roles/{id}` | `root` |
| `DELETE` | `/api/v1/roles/{id}` | `root` |
| `PUT` | `/api/v1/users/{id}/role` | `root` |
| `GET` | `/api/v1/users/{id}/environment-grants` | `root` |
| `PUT` | `/api/v1/users/{id}/environment-grants` | `root` |
| `GET` | `/api/v1/iam/policy-templates` | `read` |
| `POST` | `/api/v1/iam/policy-templates/{id}/apply` | `root` |
| `POST` | `/api/v1/iam/policy-templates/{id}/render` | `read` |
| `GET` | `/api/v1/iam/catalog` | `read` |
| `GET` | `/api/v1/iam/resources` | `read` |
| `POST` | `/api/v1/iam/resources/match` | `read` |
| `GET` | `/api/v1/iam/principals` | `read` |
| `GET` | `/api/v1/iam/principals/{type}/{id}/effective` | `read` |
| `GET` | `/api/v1/iam/simulate` | `read` |
| `GET` | `/api/v1/iam/analyze` | `read` |
| `POST` | `/api/v1/iam/policies/validate` | `read` |
| `POST` | `/api/v1/iam/preview` | `read` |
| `GET` | `/api/v1/iam/policies/{id}/versions` | `read` |
Role bodies are `{"name", "description", "abilities", "visibility"}`. Grants are `{"environment_ids": []}`.
## Next steps
Instance-wide environments and protected moves.
Decide what agents may do.
# Your account
Source: https://docs.levelrail.com/account
The Account page, your own profile, sign-in security, active sessions, API tokens and a plain-language view of what you can manage.
Settings, Account is the personal half of access: everything about your own sign-in, in one place.
## Sections
* **Profile**: your email and a display name you can edit inline.
* **Password**: change it; this signs out your other sessions.
* **Two-factor** and **Passkeys**: enrol or remove second factors.
* **Active sessions and trusted devices**: every browser signed in as you. End one, or sign out all others. Trusted devices are browsers that skip the new-device check; remove any you no longer use.
* **Linked sign-in providers**: link an enabled identity provider to your account. Linking needs your live session, so a provider can never be attached by email alone.
* **My API tokens**: how many active tokens you hold, with a link to manage them.
* **My access**: your access level, teams, attached IAM policies and, for each area you cannot manage, the permission that is missing.
## From the API
| Route | What it does |
| - | - |
| `PATCH /api/v1/auth/profile` | Change your own display name (1 to 80 characters). Session only. |
| `GET /api/v1/auth/my-access` | Your tier, abilities, teams, linked providers, attached policies and per-area capabilities. |
| `GET /api/v1/security/sessions` | Your live sessions, trusted devices and tokens. Never returns token values. |
| `DELETE /api/v1/security/sessions/{id}` | End one of your sessions. |
| `POST /api/v1/auth/sessions/revoke-others` | End every session except this one. |
Every route here is scoped to the caller. Another user's sessions read as not found unless you are an admin using `?user_id=`.
## From the CLI
```bash theme={"dark"}
levelrail-cli auth whoami --access
```
Prints your access level, abilities, teams and attached policies, then one line per area: `can` or `cannot`, with the missing ability for the latter. Add `--json` for the raw response.
# Verifying real ACME issuance against a live domain
Source: https://docs.levelrail.com/acme-verification-runbook
Verify real ACME certificate issuance against a live public domain with Let's Encrypt or another RFC 8555 CA: what to check, common failures, a recorded run.
ACME issuer configuration, settings toggles, and form validation are covered by automated tests: config-shape unit tests in `internal/ingress/acme_test.go` and end-to-end tests against a local CA in `internal/ingress/acme_live_test.go`. Those tests cannot prove issuance against a real public CA with real DNS and real reachability on ports 80 and 443. This runbook is that verification.
A first run is recorded [at the bottom of this page](#recorded-run-2026-10-05), so you do not need to repeat it before trusting the feature. Repeating it on your own host is the fastest way to confirm your firewall is right. For a version that needs no DNS, use the **Enable HTTPS** card described in [Domains and ingress](/domains-and-ingress#zero-dns-setup-sslipio-hostnames-and-one-click-https).
## Prerequisites
* A domain you control, with access to create an `A` or `AAAA` record.
* A control plane reachable from the public internet (installed with `install.sh`, as `levelrail.service`, or in Docker).
* DNS already pointing at the server's public IP and propagated.
* Ports 80 and 443 open from the internet. HTTP-01 is the check where the CA requests a file over port 80 to prove you own the domain, so a cloud firewall, a router without port forwarding or `ufw` in default-deny will break issuance while looking fine from the server itself. Wildcards need [DNS-01](/domains-and-ingress#wildcard-domains-dns-01-providers) instead.
* A `root` API token or admin dashboard access, since saving ingress settings (`PUT /api/v1/settings/ingress`) needs the `root` ability.
## Steps
Wait for propagation. From a machine outside your network, `dig +short your-domain.example.com` should return the server's public IP. Do not continue until it does: issuing before DNS propagates is the most common failure, and each failed attempt counts against the CA's rate limit.
You can also ask the control plane directly:
```bash theme={"dark"}
levelrail-cli domains check your-domain.example.com
```
Open **Domains** (Infrastructure section of the sidebar, or `/domains`) and set **Primary domain** to your hostname, for example `dashboard.example.com`. After saving, a **Certificate status** badge and a DNS check panel appear. Use the panel to confirm the platform resolves your record before enabling ACME.
Switch on **Issue real ACME certificates** and fill in:
* **Contact email**: required. The CA uses it to reach you about problems such as forced revocation. Use a monitored address.
* **Directory URL (optional)**: leave blank for Let's Encrypt production.
For a first run, set the directory URL to Let's Encrypt staging, `https://acme-staging-v02.api.letsencrypt.org/directory`. Staging certificates trigger browser warnings but exercise the same mechanism without touching production rate limits. When staging succeeds, clear the field and save again for a trusted certificate. Saving drops the previous issuer's stored certificates so the new CA re-issues them.
From the CLI:
```bash theme={"dark"}
levelrail-cli settings ingress set --primary-domain dashboard.example.com \
--acme-enabled --acme-email you@yourdomain.com
```
Saving does not issue immediately. Ingress reconciles continuously, and any routed domain qualifies for a certificate once ACME is on and an email is set, so issuance normally starts within moments. The next section says what success looks like.
## Confirm it worked
* Dashboard: the **Certificate status** badge reads **Healthy**. The issuer names your CA (`Let's Encrypt`, or an intermediate such as `R11`), never Levelrail's internal issuer. The same data is at `GET /api/v1/certificates` and `levelrail-cli domains certificates`.
* Expiry: `not_after` is about 90 days out for Let's Encrypt. A lifetime of a few hours means you are still on the internal self-signed issuer: the toggle did not save, or the page is stale.
* Renewal: nothing to configure. Caddy renews well before expiry (about 30 days out for a 90-day certificate). To see it work without waiting, look in the logs for `renewing certificate` or `certificate obtained successfully` before the current `not_after`.
* Logs: `journalctl -u levelrail -n 200 --no-pager`, or your container's logs. Look for `"logger":"tls.obtain"` and `"logger":"http.acme_client"`.
```text theme={"dark"}
"acquiring lock" -> "obtaining certificate" -> "registering account" (first
time only) -> "trying to solve challenge","challenge_type":"http-01" ->
"authorization finalized","authz_status":"valid" -> "finalizing order" ->
"successfully downloaded available certificate chains" ->
"certificate obtained successfully"
```
An error between "trying to solve challenge" and "authorization finalized" is almost always one of the DNS or port 80 failures below, not an issuer bug.
## Common failures
```mermaid theme={"dark"}
flowchart TD
A[ACME issuance failed] --> B{Challenge type in logs?}
B -->|http-01 challenge error| C{DNS resolves correctly?}
C -->|No| D["Fix DNS, wait for propagation
Run: dig +short your-domain.example.com"]
C -->|Yes| E{"Port 80/443 reachable
from internet?"}
E -->|No| F["Open ports 80, 443
Check firewall, ufw, port forwarding"]
E -->|Yes| G["Check logs for
other errors"]
B -->|rate_limit error| H["Use staging directory first
https://acme-staging-v02..."]
B -->|No challenge logged| I["Confirm domain is routed
to an active service"]
B -->|No account/cert error| J["Verify email and directory URL
Check form validation"]
```
The log shows a challenge failure such as `connection ... timed out` or `no such host` right after "trying to solve challenge".
Check `dig +short your-domain.example.com` from outside your network, wait out the record's TTL, then retry.
`dig` resolves correctly, but the log shows the challenge request timing out or being refused. Typical causes are cloud security groups, `ufw`, routers without port forwarding, and another proxy holding port 80.
Open ports 80 and 443 inbound. `levelrail-cli doctor` and the dashboard's [status page](/status-page) include reachability checks. See [Firewall: ports 80 and 443](/domains-and-ingress#firewall-ports-80-and-443).
Let's Encrypt production limits failed validations per account and hostname, and certificates per registered domain per week. Check their current limits, because they change. The log shows an explicit `rate limited` or `too many` error. Use staging for repeated attempts. Production enable attempts from the one-click flow are also capped by `APP_ACME_MAX_ATTEMPTS_PER_HOUR`.
The dashboard form rejects this before it reaches the API, and the backend rejects it the same way if you call the API directly.
Enabling ACME, or changing the directory URL (for example staging to production), drops the previous issuer's stored certificates when you save so the new CA re-issues them. If a domain still shows the wrong issuer, confirm an app actually routes that domain: ACME applies only to domains in active use by a route.
## Recorded run (2026-10-05)
Host: a 4 GB DigitalOcean droplet running the control plane from the public `install.sh`, ports 80 and 443 open to the internet, no DNS records created. Hostname: `.sslip.io` for the dashboard and for every app without a domain.
What was done and seen:
* Staging first: `POST /api/v1/settings/ingress/https` with `staging: true`. HTTP-01 validated, certificate issued by `(STAGING) ...`, shown by `GET /api/v1/certificates`.
* Production: Let's Encrypt (`acme-v02`) issued `.sslip.io` and the generated hostnames of three apps in one batch. Checked from a second server, with its normal system trust store and no `-k`: `curl` verify result `0`, issuer `Let's Encrypt` (`YE2`), `not_after` 90 days out, SAN equal to the hostname.
* `http:///` answered `308` to `https:///`.
* HSTS was enabled and disabled from the settings toggle.
What the run found and fixed:
1. Certificates were stored on disk, not in the database as this page and ADR 005 describe, so `GET /api/v1/certificates`, the certificate expiry alert rules and the issuance audit trail all saw nothing. Storage is now the database, and existing on-disk certificates are imported on first start.
2. Port 80 was closed outside ACME challenges, so any plain `http://` request was refused instead of redirected.
3. Changing the CA never took effect on a live reload, because certmagic reuses a valid certificate from any issuer. Saving now drops the other issuers' certificates.
4. `Strict-Transport-Security` was not sent on the dashboard's HTML page, only on API responses.
5. A failed issuance was invisible: only a log line. The CA's error is now captured from Caddy's certificate events and returned with a hint code.
Not covered by this run: the live capture of a CA error (item 5 is covered by unit tests only), renewal (certificates are 90 days old at best, so the renewal path ran only in tests and the `got renewal info` log lines), wildcard certificates through DNS-01, and a deliberately failing issuance against the production CA.
## See also
* [Domains and ingress](/domains-and-ingress): TLS options, DNS-01 wildcards and the one-click HTTPS flow.
* [Installing Levelrail](/installing): initial setup and prerequisites.
* [Troubleshooting](/troubleshooting): broader platform diagnostics.
* [ADR 005: Caddy embedded ingress](https://github.com/glincker/levelrail/blob/main/adr/005-caddy-embedded-ingress.md): the design decisions.
# Agent tooling audit
Source: https://docs.levelrail.com/agent-tooling-audit
Size, overlap and annotation audit of the levelrail-mcp tool surface, with the token budget test that keeps it from regressing.
An MCP client loads every tool definition from `tools/list` into the model's context before the first user message. This page records what that costs for `levelrail-mcp` in each mode, which tools overlap, and the tests that stop the listing growing unnoticed.
Numbers are estimates: the serialized `tools/list` JSON length divided by 4 (no tokenizer is in `go.mod`). They are good for comparing modes and catching regressions, not for billing. They are a snapshot of the 156 registered tools and drift as tools are added, so rerun the commands below for current figures.
[MCP tool surface](/mcp-tool-surface) counts each tool's name, description and input schema per toolset. This page measures the whole serialized `tools/list` response a client receives, which also carries output schemas and annotations, so its totals are larger.
## Tools and tokens per mode
Measured by `go test -run TestToolListTokenBudget -v ./internal/mcptools` (or `scripts/mcp-token-budget.sh`).
| Mode | Tools | Estimated tokens | Test budget |
| - | - | - | - |
| `read-only` | 118 | about 46,200 | 47,000 |
| `standard` (default) | 146 | about 61,200 | 62,000 |
| `full` | 156 | about 66,700 | 68,000 |
Modes are defined in `internal/mcptools/modes.go` and driven by the class table in `internal/mcptools/classes.go`: 118 read, 28 mutating and 10 destructive tools. `read-only` registers read tools, `standard` adds mutating tools, and `full` adds destructive ones. The API token's own abilities still apply on top of the mode.
## The agent-core profile
The `agent-core` profile lists 15 tools and omits output schemas from `tools/list` (results are still returned as text and structured content). Its full listing is about 2,600 estimated tokens against about 66,700 for `full` mode. The tools are listed in [MCP tool surface](/mcp-tool-surface#agent-core-profile). A test holds the listing under a budget (default 3,500, override with `APP_MCP_TOKEN_BUDGET_AGENT_CORE`).
Use this profile for autonomous agents, which rarely need more than these 15 tools: `levelrail-mcp --tool-profile agent-core` or `APP_MCP_TOOL_PROFILE=agent-core`.
## Overlapping tools
| Tools | Overlap | Guidance |
| - | - | - |
| `get_app_status`, `list_deploys` | Identical handler and result (current reconcile conditions). | `agent-core` includes both; in a custom set, one is enough. |
| `deploy_app`, `rollback_app` | Same request and handler. | Both kept: intent is useful and the classes differ. |
| `list_deploys`, `list_deploy_attempts`, `list_deployments`, `list_failed_deploys` | Four ways to list deploy history at different scopes (one app, fleet, failures only). | Prefer `list_deploys` or `list_deploy_attempts` in compact setups. |
| `get_resource_recommendation`, `get_database_resource_recommendation` | Same shape for apps and databases. | Separate by design. |
| `get_app_logs`, `get_model_logs`, `list_archived_logs`, `query_logs` | Four log entry points. | `query_logs` returns a capped, filtered excerpt and is the one in `agent-core`. |
| `preview_promote_app` and `promote_app`, `preview_clone_environment` and `clone_environment` | Preview and apply pairs. | By design: call the read-only preview first. |
## Annotations
* Every registered tool carries a title, `readOnlyHint`, `openWorldHint` and, for non-read tools, `destructiveHint`. `TestEveryToolClassified` in `internal/mcptools/modes_test.go` fails the build if a tool is unclassified.
* `idempotentHint` is set only for tools whose name starts with `set_`. Tools such as `approve_*`, `reject_deploy_approval`, `expire_alert_silence` and `restart_app` are also safe to repeat but are marked non-idempotent.
* Sensitive and untrusted-output tools carry `_meta` flags (`levelrail/sensitive` and its untrusted counterpart), set from the class table.
## Where tokens go
Output schemas take up most of the space. Typed handlers derive them from the Go result struct, so the same large result schema repeats across tools (for example the app resource in `get_app`, `list_apps` and `restart_app`, and the deploy result in `deploy_app`, `rollback_app` and `approve_deploy_approval`). Input schemas and descriptions are the next largest parts. The cheapest fix is a smaller listing, which is what the `agent-core` profile and the `APP_MCP_TOOLSETS` filter give you.
## Regression tests
`internal/mcptools/budget_test.go` serializes `tools/list` for each mode over an in-memory MCP session and fails when the estimate exceeds that mode's budget.
| Env var | Effect |
| - | - |
| `APP_MCP_TOKEN_BUDGET_READ_ONLY` | Override the `read-only` ceiling (estimated tokens). |
| `APP_MCP_TOKEN_BUDGET_STANDARD` | Override the `standard` ceiling. |
| `APP_MCP_TOKEN_BUDGET_FULL` | Override the `full` ceiling. |
| `APP_MCP_TOKEN_BUDGET_AGENT_CORE` | Override the `agent-core` profile ceiling. |
Run it with a report of the heaviest tools:
`scripts/mcp-token-budget.sh`
Raise a budget only with a reason in the PR that changes it.
# Working with AI agents
Source: https://docs.levelrail.com/agents
Connect an AI coding agent to your instance, scope its token, and give it the instructions it needs to deploy safely.
An agent talks to the platform the way you do: through the CLI (`levelrail-cli`) or the MCP server (`levelrail-mcp`). Everything it does goes through the normal API with the token you give it, so its access is exactly that token's abilities, and every change lands in the audit log under the agent's name.
## Set up a project with `init`
Run this in the project root:
```bash theme={"dark"}
levelrail-cli init
```
It detects the stack and writes three files:
| File | What it is |
| - | - |
| `app.yaml` | The app spec, validated with the same parser the control plane uses before it is written. |
| `AGENTS.md` | Instructions for an agent: deploy, wait, read a failure, roll back, set env and secrets safely. Uses the CLI name you ran and your API URL when the CLI knows it. |
| `.mcp.json` | A stdio MCP server entry. The token is an env var reference (`${APP_API_TOKEN}`), never a value. |
Detection order: Dockerfile, compose file, `package.json` (Next.js, NestJS, Express, Fastify, Vite), `go.mod`, Java (`pom.xml`, Gradle), Python (`requirements.txt`, `pyproject.toml`, `Pipfile`), then a plain `index.html` static site. Provider names match the build system's own, so what `init` reports lines up with what a deploy will detect. Nothing from the project is executed.
Flags:
* `--dry-run` shows the plan and changes nothing.
* `--force` overwrites files that already exist. Without it existing files are skipped, and the diff of what would change is shown.
* `--yes` skips the confirmation prompt. A script without a terminal must pass `--yes` or `--dry-run`.
* `--json` prints the plan and result as one JSON value.
* `--mode` picks the tool exposure written to `.mcp.json`: `agent-core` (default), `read-only`, `standard` or `full`.
* `--dir`, `--api-url`, `--profile` and `--mcp-binary` override the directory, the URL written into the files, the credentials profile and the MCP binary name.
## Install the skill
The repository ships an agent skill that covers the deploy, wait, diagnose and roll back loop, so an agent follows the safe path without you pasting in instructions:
```bash theme={"dark"}
npx skills add glincker/levelrail
```
`levelrail-cli init` writes the same guidance into a project's own `AGENTS.md`, filled in with that project's app name and API URL.
## Modes
The MCP server exposes a subset of its tools depending on the mode. Token counts are estimates of the `tools/list` payload the client loads into context.
| Mode | Tools | Estimated tokens | Use it for |
| - | - | - | - |
| `agent-core` (a tool profile, `--tool-profile agent-core`) | 15 | about 2,600 | Shipping and debugging one app. The default for `init`. |
| `read-only` | 118 | about 46,200 | Observers that must not change anything. |
| `standard` | 146 | about 61,200 | Read and mutating tools, no destructive ones. |
| `full` | 156 | about 66,700 | Every tool, including destructive ones. |
See [Agent tooling audit](/agent-tooling-audit) for how the numbers are measured.
## Tokens
Give each agent its own token so its actions are attributable and it can be revoked alone.
```bash theme={"dark"}
levelrail-cli tokens create --name ci-agent --preset deployer --agent "Claude Code"
```
`--preset` takes `observer`, `deployer` or `operator` (see the table below); `--abilities` sets an explicit list instead.
The `--agent` label is shown on the tokens page and recorded on every audit entry the token makes. Filter the audit log by it with `audit-log --agent NAME` or the agent chip on the audit log page. Settings > Agents in the dashboard shows the MCP connect snippet (copyable, per mode), lists agent tokens with their last used time, and has a "Create agent token" flow with the same three presets:
| Preset | Abilities | Can do |
| - | - | - |
| Read-only observer | `read` | Look at apps, logs, metrics and deploys. |
| Deployer | `read`, `deploy` | Also trigger deploys and rollbacks. |
| Full operator | `read`, `read:sensitive`, `write`, `write:sensitive`, `deploy` | Also change config, env vars, domains and secrets. Never `root`. |
The token is shown once, when it is created. Store it in your shell profile or secret manager as `APP_API_TOKEN`, not in the repo.
## Read logs by level
The app Logs page (Search tab) shows a chip per level (error, warn, info, debug) with the number of loaded lines at that level, and a "Showing N of total" line. Each row has a gutter with an icon and the level word, so the level does not depend on color. The search loads the newest 1,000 matches; the total counts every match in the range. Agents get a capped excerpt of the same data from `levelrail-cli logs query` or the MCP `query_logs` tool, which filter by `level`, time window and text.
## Dry run before changing anything
* `plan_change` (MCP) previews a single mutating tool call without running it, and `plan_apply` previews a resource file. `levelrail-cli apply --dry-run` is the CLI form.
* `levelrail-cli apps preflight NAME` checks DNS, ports, disk, the image and required env before a deploy.
## Wait for a deploy
An agent should never assume a deploy worked. `wait_for_deploy` (MCP) or `levelrail-cli apps wait NAME` blocks until the deploy converges and reports success or the structured failure. On failure, read it with `levelrail-cli apps deploys show NAME DEPLOY_ID`, fix one thing, and retry. Roll back first if production is down.
## Next steps
Run `levelrail-mcp` over stdio or HTTP and scope its token.
Token cost per mode and the budget tests.
# AI assistant integration: levelrail-mcp
Source: https://docs.levelrail.com/ai-assistant
Run levelrail-mcp over stdio for a local client or over the network for a remote MCP assistant, and how to scope the API token it uses.
![Anthropic]( "Anthropic")
![OpenAI]( "OpenAI")
`levelrail-mcp` is an MCP (Model Context Protocol) server that turns the control plane's REST API (`/api/v1`) into MCP tools. It is a thin client like `levelrail-cli`: it authenticates with an API token and calls the same routes the CLI and dashboard do. It has no special internal access, and a token with fewer abilities than a tool needs gets the same 403 the REST API would return.
Source: `cmd/levelrail-mcp` and `internal/mcptools`.
## What it exposes
There is one tool per REST capability: apps, deploys, databases, nodes, domains, certificates, backups, templates, registry credentials, IAM, organizations, alerts, metrics, diagnostics, [feature flags](/feature-flags) and more. See the [API reference](/api-reference) for the underlying routes, and [MCP tool surface](/mcp-tool-surface) for the tool count per toolset.
Notable read-oriented tools for day-2 operation:
* `get_attention`: everything that needs attention now (failing apps, offline nodes, bad certificates, doctor warnings), critical first. Same list as `levelrail-cli attention`.
* `get_node_status_history`: a node's recent status transitions (`GET /api/v1/nodes/{id}/events`).
* `list_audit_log`: supports `q` (text search) and `status=failed` to find rejected requests.
* `list_control_plane_backups` and `create_control_plane_backup`: snapshots of the control plane's own database. The create tool writes a backup file, so give an assistant that should not do that a `read`-only token.
Env tools:
* `set_app_env` and `unset_app_env`: change one plain env var or, with `secret: true`, one write-only secret (same routes as `levelrail-cli apps env` and `apps secrets`). Secret values are never returned or logged. The result names the key and says whether a redeploy is needed; follow with `deploy_app` or `restart_app`.
## Two ways to run it
This is the default and needs no extra flags. An MCP client that can launch local subprocesses (Claude Desktop, Claude Code and similar tools) starts `levelrail-mcp` itself and talks to it over stdin/stdout:
```json theme={"dark"}
{
"mcpServers": {
"levelrail": {
"command": "levelrail-mcp",
"args": [],
"env": {
"APP_API_TOKEN": "",
"APP_API_URL": "http://localhost:8080"
}
}
}
}
```
If `levelrail-cli` is already configured on the same machine (`levelrail-cli auth login` writes `~/.config/levelrail-cli/credentials`), `levelrail-mcp` picks up the same token and URL automatically and the `env` block above can be omitted.
No incoming authentication is required in this mode: only a local process on the same machine can spawn a stdio subprocess.
Some MCP clients cannot spawn a local subprocess, for example an AI assistant that runs as its own independently deployed service. For that case, run `levelrail-mcp` with `--transport=http` to serve the MCP Streamable HTTP transport over the network instead:
```bash theme={"dark"}
levelrail-mcp --transport=http --listen=127.0.0.1:8090 --token ''
```
* Binds to loopback by default (`127.0.0.1:8090`). Pass `--listen` explicitly (for example `0.0.0.0:8090`, or a WireGuard-mesh address) to expose it beyond the local machine. You have to opt in.
* Fails closed with no token. A network listener is reachable by anything that can route to it, so `--transport=http` with no API token configured (via `--token`, `APP_API_TOKEN`, or a `levelrail-cli auth login` credentials file) refuses to start rather than listen unauthenticated.
* Requires the same bearer token on every incoming request. The token this process already uses against the control plane's REST API is the one an MCP client must present as `Authorization: Bearer ` on every request to the listener. There is no separate auth to configure.
* Put a reverse proxy or the WireGuard mesh (see [Multi-node](/multi-node)) in front of it for TLS if the client is not on the same trusted network; `levelrail-mcp` itself speaks plain HTTP.
## Modes and toolsets
Every tool carries MCP annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`, and a title), derived from one classification table in `internal/mcptools/classes.go`. Tools that touch credentials or command output also carry `_meta["levelrail/sensitive"] = true`. Annotations are hints only, so the server enforces the same split at registration time.
`APP_MCP_MODE` (or `--mode`) chooses which classes are registered at all. A tool that is not registered costs no model context and cannot be called:
| Mode | Registers |
| - | - |
| `read-only` | read tools only (get, list, explain, diagnose, compare, preview) |
| `standard` (default) | read and mutating tools (deploy, restart, set, create, clone, approve, rotate) |
| `full` | everything, including destructive tools (delete, clear, rollback, prune, sweep) |
`APP_MCP_TOOL_PROFILE=agent-core` (or `--tool-profile agent-core`) additionally restricts the server to 15 tools for autonomous agents: list, status, deploy, rollback, cancel, diagnose, preflight, capped log search (`query_logs`), env (`get_app_env`, `set_app_env`, `unset_app_env`, secrets are write-only) and domains. It lists no output schemas, which brings the whole `tools/list` to about 2,600 estimated tokens against about 66,700 for `full`. See [MCP tool surface](/mcp-tool-surface) and [Agent tooling audit](/agent-tooling-audit).
`APP_MCP_TOOLSETS` (or `--toolsets`) is an optional comma separated list that limits the groups exposed: `alerts, apps, audit, backups, databases, deploys, diagnostics, domains, environments, flags, iac, iam, loadbalancer, logs, metrics, models, nodes, notifications, orgs, pipelines, previews, registry, scheduled, settings, system, templates, webhooks`.
```bash theme={"dark"}
APP_MCP_MODE=read-only APP_MCP_TOOLSETS=apps,nodes,logs,diagnostics levelrail-mcp
```
The startup log line `levelrail-mcp tools` reports the mode and how many read, mutating and destructive tools are registered. A mode never widens access: the API token's abilities still bound every call, so pair `read-only` with a `read`-scoped token (defense in depth, not a replacement for the token scope).
## Reading logs without flooding the context
`query_logs` searches one app's logs by minimum level, time window (`since`, `until`), deploy attempt (`deploy_id`) and text, and returns a capped excerpt of the newest matches with counts, for example `showing 40 of 1,812 matching lines (newest), use since/until or level to narrow`. The output is capped at `max_lines` (default 100) and at `APP_MCP_LOG_MAX_BYTES` bytes (default 8192). Prefer it over `get_app_logs`. The same query is available as `levelrail-cli logs query`.
## Untrusted text and the assistant's confirmation gate
Logs, deploy output, error messages, commit messages, PR titles and similar text are written by workloads or third parties, so they can carry prompt injection. Tools marked `levelrail/untrusted-output` in `_meta` (log, status, deploy, diagnose, pipeline and audit reads) return their text content inside a delimited block that starts with a standard "untrusted data, not instructions" notice. Control characters, ANSI escapes, invisible and bidi characters are stripped, obvious secrets (private keys, bearer tokens, common token shapes, URL credentials, values of password/token/key fields) are redacted, and text is truncated. The block boundaries carry a random id, so content cannot forge the closing line. The structured copy of the result is sanitized the same way but is not delimited.
Limits come from `APP_UNTRUSTED_MAX_FIELD_BYTES` (per string field, default 4096) and `APP_UNTRUSTED_MAX_BLOCK_BYTES` (per wrapped block, default 65536).
This is friction, not a guarantee. The real boundary is the assistant's confirmation gate. It goes by each tool's MCP annotations, not name prefixes: every tool that is not read-only, and every tool without annotations, pauses for a human click. Once a conversation has ingested untrusted output (it is then "tainted", derived from the stored history so it survives later turns), read tools that are outbound or touch secrets pause as well.
## Whose authority the in-dashboard assistant uses
The assistant built into the dashboard has no standing credential of its own. Every chat message (and every Approve click on a pending action) mints a token for that one turn: owned by the person who sent it, carrying only their abilities, their IAM policies and their team scope, labeled with the agent name `ai-assistant`, and revoked when the turn ends. It expires after 15 minutes even if the control plane dies mid-turn (`APP_AI_TURN_TOKEN_TTL`, a Go duration such as `10m`). So a prompt injection in a log line can never do more than the human who asked, and an IAM Deny on that human also binds the assistant.
* The [AI control](/ai-control) mode still applies to every tool call, because the turn token is an agent token.
* Each tool call row in the chat shows who it ran as and which abilities the API checked for that call, for example "Acting as [alice@example.com](mailto:alice@example.com) with read". `levelrail-cli ai chat` and `ai sessions show` print the same.
* Audit entries made by a turn token carry `on_behalf_of` (`user:`) next to the `ai-assistant` agent name.
* Chat sessions belong to the principal that created them. Another user or token, even one holding `root`, gets 404 for them. Sessions created before this release have no owner and are no longer listed.
## Generating and scoping a token
Use the same token machinery the CLI and dashboard already use ([Identity and access](/identity-and-access#api-tokens)): scoped, revocable bearer credentials minted with the six-string ability vocabulary (`read`, `read:sensitive`, `write`, `write:sensitive`, `deploy`, `root`).
```bash theme={"dark"}
levelrail-cli tokens create --name "ai-assistant" --abilities read,deploy
```
`--preset observer|deployer|operator` is a shortcut for common sets. Pick the narrowest ability set the assistant actually needs. A read-only assistant that only diagnoses and reports needs `read` (add `read:sensitive` only if it must see secrets or env values); one that can also trigger rollbacks or redeploys needs `deploy` too. Every request through `levelrail-mcp`, over either transport, is attributed to the `mcp` client kind in the audit log (`GET /api/v1/audit-log`), so scoped-down tokens are traceable the same way CLI and dashboard activity already is.
### Dry runs with plan\_change
`plan_change` previews a mutating tool call without running it. Pass `tool` and the `arguments` you would use; it returns `executed: false`, a summary, the before and after of each changed field, and `blocked` with `blockers` when a freeze window or a required approval would stop the call now. It uses reads only (plus the API's own dry runs for `bulk_apps`, `bulk_delete_apps`, `apply_resources` and `promote_app`), so it is safe for a read-only token.
Planned tools: `deploy_app` and `rollback_app` (image and digest, freeze window), `set_app_env` and `unset_app_env` (key level only, values are never shown), `set_app_domains`, `restart_app`, `cancel_deploy`, `promote_app` (the promotion preview with its blockers), `apply_resources` (the resource plan and its hash), and the two bulk tools (the target list). Every other mutating tool answers `plannable: false` and is not run, so an agent knows to ask the user before calling it. Mutating tools do not accept a `dry_run` argument (the call is rejected before it reaches the API), except the bulk tools, which keep their own.
### Agent identity
Label a token as issued to an agent and every audit entry made with it records who it was:
```bash theme={"dark"}
levelrail-cli tokens create --name ci-agent --abilities read,deploy --agent deploy-bot --agent-description "ships main from CI"
levelrail-cli audit-log --agent deploy-bot
```
`--agent` sets the agent name (at most 64 characters, description at most 500). Audit entries from that token carry `agent_name`, and when the caller is `levelrail-mcp` also `agent_client`, the MCP client name and version it reported at initialize (self-reported, informational). `GET /api/v1/audit-log?agent=NAME`, the `list_audit_log` MCP tool's `agent` argument and `audit-log --agent` filter on it, and `tokens list` shows an AGENT column. The label changes nothing about what the token may do: a read-only agent token still cannot deploy, and that rejected request is not an audit entry (only permitted requests are recorded).
## Running a separately licensed AI assistant
`levelrail-mcp` works with any MCP-compatible AI assistant, including one licensed under terms (for example AGPL) that are incompatible with this project's Apache 2.0 license, or any third-party assistant you do not want coupled to this codebase.
That assistant must always run as its own, separately deployed service. It connects to Levelrail purely as an MCP client, over stdio (if co-located and it can spawn a subprocess) or over `--transport=http` (if it runs elsewhere). Its code must never be imported, vendored, copied into, or otherwise combined with this repository, regardless of transport. An integration is a boundary, not a merge.
## Troubleshooting
### MCP server fails to start with "cannot read credentials"
Likely cause: No API token is configured, or the token file is missing.
How to confirm: Check if credentials exist: `test -f ~/.config/levelrail-cli/credentials && echo found || echo missing`. Check the environment: `echo $APP_API_TOKEN` to see if a token is set there.
Fix: Either set the `APP_API_TOKEN` env var to your token, or run `levelrail-cli auth login --device` on the control plane machine to create a credentials file. The MCP server will pick it up on the next start.
Still stuck: See [API tokens](/identity-and-access#api-tokens) to create and manage tokens.
### MCP HTTP transport listening fails with "bind: permission denied"
Likely cause: You are trying to bind to a port below 1024 (which requires root), or the port is already in use.
How to confirm: Check what is using the port: `lsof -i :` or `ss -tlnp | grep `. Try a port above 1024: `levelrail-mcp --transport=http --listen=127.0.0.1:8090`.
Fix: Either use a port above 1024 (recommended), or run the command with `sudo`. If the port is already in use, pick a different port.
Still stuck: Run the server on localhost (127.0.0.1) instead of 0.0.0.0 to avoid needing high ports.
### MCP client cannot connect to remote server, "connection refused"
Likely cause: The MCP server is not running, or it is not listening on the address the client is using.
How to confirm: Check if the server is running: `curl http://:` should fail gracefully (not connection refused). Verify the listening address: look at the startup logs for `levelrail-mcp listening on` or check with `ss -tlnp | grep 8090`.
Fix: Start the MCP server if it is not running: `levelrail-mcp --transport=http --listen=0.0.0.0:8090` (or use a specific IP). Verify the client is using the same address and port. If the server is behind a firewall, ensure the port is open from the client's network.
Still stuck: Use `--listen=0.0.0.0:` to listen on all interfaces (if secure), or put the server behind a reverse proxy with TLS.
### Tool execution returns untrusted output warning and no results
Likely cause: The tool's output contains untrusted data (logs, error messages, etc.), and the MCP server wrapped it with a safety notice.
How to confirm: The output will start with something like `` and contain a warning. The structured data under `_meta["levelrail/untrusted-output"]` is also sanitized.
Fix: This is intentional: the assistant must confirm it has read untrusted output before acting on it. The data is safe to read, only flagged as coming from an untrusted source (user logs, deployment output, etc.). The assistant's confirmation gate will pause before executing any action based on this data.
Still stuck: Pass a read-only token to the MCP server to prevent tool calls that might act on tainted data.
### Agent identity not appearing in audit log
Likely cause: The token was not created with the `--agent` flag, or the server is not using that token.
How to confirm: Check the token's agent field: `levelrail-cli tokens list | grep | awk '{print $NF}'` should show the agent name. Check the audit log: `levelrail-cli audit-log --limit 10 | grep agent` should show entries with agent\_name.
Fix: Create a new token with agent identity: `levelrail-cli tokens create --name my-bot --agent bot-name --abilities read,deploy`. Use that token in the MCP server.
Still stuck: See [Agent identity](/ai-assistant#agent-identity) for the token creation syntax.
## See also
* [ai-assistant-chat.md](/ai-assistant-chat): the in-app dashboard/CLI chat, a separate feature built on this same engine and confirmation gate, behind the `ai-chat` experimental flag.
* [identity-and-access.md](/identity-and-access): API tokens, abilities, and the audit log.
* [api-reference.md](/api-reference): every REST route the MCP tools wrap.
* [cli-reference.md](/cli-reference): `tokens create`/`list`/`revoke` and `auth login`.
# In-app AI assistant chat
Source: https://docs.levelrail.com/ai-assistant-chat
The in-app AI assistant chat with your own key for Anthropic, OpenAI compatible, Gemini or self-hosted models, how to enable it behind the ai-chat experimental flag, and the confirmation gate that bounds what it can do.
This is a chat panel in the dashboard, backed by the control plane's own `internal/ai` engine. It reads your apps, deploys, logs, metrics and diagnostics through the same tool surface the MCP server exposes, and it can propose actions. Every action that would change anything pauses for an explicit confirmation click first. AI in Levelrail only reads and suggests on top of the API, and this chat uses the same engine and confirmation gate described in [AI assistant integration](/ai-assistant#untrusted-text-and-the-assistants-confirmation-gate), as a dashboard panel and a CLI command.
This is separate from `levelrail-mcp`. To point an external MCP client (Claude Desktop, Claude Code, your own) at this control plane, read [AI assistant integration](/ai-assistant) instead.
Source: `internal/ai`, `internal/api/ai_chat.go`, `internal/api/ai_settings.go`, `internal/api/ai_providers.go`, `web/src/components/AiChatPanel.tsx`, `cmd/levelrail-cli/ai.go`.
## Quick start in 3 steps
```bash theme={"dark"}
APP_EXPERIMENTAL=ai-chat
```
Restart the control plane. The **AI assistant** page now appears in the dashboard.
CLI: `levelrail-cli settings ai-assistant providers set anthropic --api-key-env ANTHROPIC_API_KEY` then `levelrail-cli settings ai-assistant use anthropic claude-sonnet-5-5`
Dashboard: Settings > AI assistant, pick a provider and paste its key.
Dashboard: **AI assistant** in the nav. Ask "why did my last deploy fail?" or "show me the errors in logs today". Approve actions if prompted.
## Enabling it
Two things are required.
The chat is gated behind `ai-chat` (see [Experimental features](/experimental-features)). Set this on the control plane:
```bash theme={"dark"}
APP_EXPERIMENTAL=ai-chat
```
With the flag off (the default), every route under `/api/v1/ai/sessions` and `/api/v1/settings/ai-assistant` answers `404`, the dashboard hides the **AI assistant** nav entry and the `/ai-assistant` page, and the CLI hides `ai chat` and `ai sessions` from `--help` and completion.
Nothing runs until you bring your own key. Pick a provider from the list under [Providers](#providers), save its key, then choose the active provider and model. The key is checked with the provider before it is stored, kept with the same envelope encryption as other secrets, and never returned, logged, or written to the audit log.
```bash theme={"dark"}
export OPENAI_API_KEY=sk-...
levelrail-cli settings ai-assistant providers set openai --api-key-env OPENAI_API_KEY
levelrail-cli settings ai-assistant providers models openai
levelrail-cli settings ai-assistant use openai gpt-4.1
```
`levelrail-cli settings ai-assistant get` reports the active provider, the fallback chain and whether a key is configured.
Over the API, `GET /api/v1/settings/ai-assistant` returns `configured`, `active`, `fallback`, `limits` and `token_usage_month`, and `PUT` takes `active` and/or `fallback` (each `{provider, model}`). Keys are saved per provider with `PUT /api/v1/settings/ai-assistant/providers/{id}` and removed with `DELETE` on the same path.
Open **Settings → AI assistant**, pick a provider, paste its key, then choose the active model.
With both set, the **AI assistant** page appears in the dashboard and `levelrail-cli ai chat` stops returning the experimental-disabled error.
## What it can do
* Read, always: logs, metrics, deploy history, node status, certificates, and crashloop diagnosis, through the read-only tool classification described under [Modes and toolsets](/ai-assistant#modes-and-toolsets). These run automatically, with no confirmation, the moment the model asks for them.
* Propose, never execute directly: a mutating tool call (deploy, rollback, restart, an env change, and so on) is recorded as a pending confirmation and streamed to the UI as a `tool_call_proposed` event. It does not run until a human clicks **Approve** (dashboard) or runs `levelrail-cli ai sessions resolve --approve` (CLI). Rejecting it is just as explicit a path (`--reject`), and the model sees "User declined to run this action" as the result either way, so it can explain and move on rather than retry blindly.
* Never in the reconciliation path: the assistant only calls the platform's own REST API (`internal/apiclient`), the same path any external MCP client or the CLI uses. It has no privileged entry point into the reconciler, the Docker client, or the database.
Untrusted-text handling and the confirmation gate's exact rules (what counts as tainted, which annotations make a tool auto-run) are documented once, under [Untrusted text and the assistant's confirmation gate](/ai-assistant#untrusted-text-and-the-assistants-confirmation-gate).
## Using it
### Dashboard
**AI assistant** in the global nav (hidden unless the flag is on) opens a chat panel: a message composer, a transcript with the assistant's replies rendered as markdown (code blocks, lists, links; raw HTML and images are stripped, since a reply can echo untrusted tool output), and a confirmation card with Approve/Reject buttons for any pending mutating tool call. **History** lists past sessions so you can resume or delete one; **New chat** starts a fresh one.
### CLI
```bash theme={"dark"}
# One-shot: start a session, send a message, stream the reply to stdout.
levelrail-cli ai chat "why did the last deploy of web fail?"
# Continue the same conversation (the first run printed its session id to stderr).
levelrail-cli ai chat "roll it back then" --session aisess_abc123
# A mutating call proposed above pauses; approve or reject it explicitly.
levelrail-cli ai sessions resolve aisess_abc123 aiconf_xyz789 --approve
# Manage sessions.
levelrail-cli ai sessions list
levelrail-cli ai sessions get aisess_abc123
levelrail-cli ai sessions delete aisess_abc123
```
`--json` on any `ai` subcommand switches to one JSON object per stream event (or the plain resource for `sessions list`/`get`/`delete`), the same `--json`/`--output`/`--query` convention every other `levelrail-cli` command follows.
### REST API
`POST /api/v1/ai/sessions`, `GET /api/v1/ai/sessions`, `GET`/`DELETE /api/v1/ai/sessions/{id}`, `POST /api/v1/ai/sessions/{id}/messages` (Server-Sent Events: `text_delta`, `tool_call_proposed`, `tool_result`, `error`, `done`), and `POST /api/v1/ai/sessions/{id}/confirmations/{confirmation_id}` (same SSE framing, for resolving a pending call). All of it requires `AbilityRoot`: a confirmed message can execute any mutating tool the platform exposes once approved, the same blast radius a root-scoped token would need to reach those actions directly, so do not hand this surface to a lower-privileged token.
## Scope and limits
* Single admin user, no per-user session isolation beyond the root-only gate: anyone who can authenticate as root can see and act on any session.
* One active provider and model at a time, plus an ordered fallback chain used only for transient failures. There is no per-request model routing.
* Not yet covered by real-infrastructure testing: `test/e2e/ai_chat_test.go` proves the full session lifecycle (create, send a message with an auto-executed read tool call, get, list, delete, and the flag-off 404) over real HTTP against a real router and a real `ai.Engine`, but with a scripted fake `ai.Provider`, never a live call to a provider's API. See [Feature status](/feature-status) for the current evidence tier.
## Providers
Three adapters cover every provider. All are native Go over `net/http`, with no vendor SDK, so keys stay on the control plane and never reach the browser.
| Adapter | Providers |
| - | - |
| `anthropic` | Anthropic (Messages API, tool use, streaming, prompt caching) |
| `openai_compat` | OpenAI, Azure OpenAI, OpenRouter, Groq, Together AI, Mistral, DeepSeek, xAI, Fireworks AI, Ollama, vLLM, LM Studio, and "Other (OpenAI compatible)" for any server speaking Chat Completions |
| `gemini` | Google Gemini (native `generateContent`) |
`levelrail-cli settings ai-assistant providers list` shows each provider, whether a key is saved and the result of its last test. Self-hosted providers (Ollama, vLLM, LM Studio, Other) can be saved without a key. Azure OpenAI needs a base URL (your resource's endpoint) and a key; it authenticates with the `api-key` header.
The model is a validated choice: `providers models ` lists what the provider offers (cached for `APP_AI_MODELS_CACHE_TTL`, default 10m), and you can still type any id the provider did not list. Default models in the catalog are suggestions only.
### Self-hosted and private endpoints
A base URL is checked before anything is sent to it:
* Public endpoints must be `https`, so a key is never sent in clear text.
* Every connection is checked against the resolved IP at dial time, so a public hostname that later resolves to a private address (DNS rebinding) is refused, as is a redirect to a different host.
* Private, loopback and link-local addresses (Ollama on `localhost`, vLLM on a LAN address) are allowed only for `openai_compat` providers, only for a root caller, and only for the host that caller typed in. Saving such a URL writes an `ai_assistant.private_base_url` audit entry with the provider and host (never the key).
* Credentials inside the URL are rejected.
### Example: Ollama on a Levelrail GPU node
Deploy the model as described in [AI models](/ai-models), then point the assistant at it:
```bash theme={"dark"}
levelrail-cli models deploy --name chat --engine ollama --model llama3.1:8b
levelrail-cli settings ai-assistant providers set ollama --base-url http://:11434/v1
levelrail-cli settings ai-assistant providers models ollama
levelrail-cli settings ai-assistant use ollama llama3.1:8b
```
Pick a model that supports tool calling; a model without it can chat but cannot read your apps. If the control plane cannot reach the node address, `providers set` answers with an `unreachable` error and nothing is saved. A hosted `levelrail` model provider is not built in: use this form with the model's endpoint instead.
## Fallback chain
```bash theme={"dark"}
levelrail-cli settings ai-assistant fallback set groq:llama-3.3-70b-versatile anthropic:claude-haiku-5-5
```
When the active provider fails with a rate limit, an overload, a timeout or a network error, or its account is out of quota, the next entry is tried. It never falls back on an invalid key, a content refusal, a too-long conversation or an unknown model, and never after the answer has started streaming, so output is not duplicated. Each provider is also retried first with jittered backoff that honors `Retry-After`. At most 5 entries. Removing a provider's key removes it from the chain.
## Usage and budget
Every model call records prompt and completion tokens, provider, model, latency and error class. `levelrail-cli settings ai-assistant usage --days 30` (or `GET /api/v1/ai/usage?days=30`) aggregates by day, provider and model. Set `APP_AI_MONTHLY_TOKEN_BUDGET` to block new turns once the calendar month's tokens reach it; the default is no budget. A blocked turn is not stored, and the chat shows how many tokens were used.
## Error meanings
| Code | Meaning | Fix |
| - | - | - |
| `invalid_key` | The provider rejected the key. | Save a valid key. |
| `quota_exhausted` | Out of credit or quota. | Add credit, or add a fallback. |
| `rate_limited` | Too many requests. | Wait, or add a fallback. |
| `overloaded` | Provider overloaded or down. | Retry, or add a fallback. |
| `context_too_long` | Conversation exceeds the model's window even after trimming. | Start a new chat or pick a larger model. |
| `model_not_found` | The provider has no such model. | Pick one from `providers models`. |
| `network` | Provider unreachable or timed out. | Check outbound network and base URL. |
| `stream_interrupted` | The connection dropped mid-answer. | Send the message again. |
| `blocked_address` | The URL failed the address policy. | See the endpoint rules above. |
| `budget_exhausted` | Monthly token budget used. | Raise the budget or wait. |
The chat stream sends an `error` event with `code`, `message`, `fix`, `provider` and `retryable` just before `done`. A cut-off answer is kept in the session marked as interrupted, and a tool call that never finished streaming is dropped, never run.
## Harness limits
| Variable | Default | Effect |
| - | - | - |
| `APP_AI_STEP_TIMEOUT` | `120s` | Longest a single model call may take. |
| `APP_AI_MAX_STEPS` | `12` | Model calls per turn before it stops. |
| `APP_AI_TOOL_OUTPUT_MAX_BYTES` | `32768` | Cap per tool result; cut output ends with `[truncated N bytes]`. |
| `APP_AI_DEFAULT_CONTEXT_TOKENS` | `128000` | Window assumed when the provider does not report one. |
| `APP_AI_MONTHLY_TOKEN_BUDGET` | none | Monthly token budget. |
| `APP_AI_RETRY_MAX_ATTEMPTS`, `APP_AI_RETRY_BASE_DELAY`, `APP_AI_RETRY_MAX_DELAY` | `3`, `1s`, `30s` | Retry policy. |
| `APP_AI_VALIDATE_TIMEOUT` | `10s` | Key check and model list timeout. |
Before each call the oldest tool results are replaced with a placeholder, then the oldest whole exchanges are dropped, until the conversation fits the model's window. The system prompt and your latest message are never dropped. Tool arguments are checked against the tool's schema before it runs, and a malformed or unknown call goes back to the model as an error. Whatever the provider, any tool that is not known read-only still waits for your approval.
## What is unverified
Everything here was tested against fake HTTP servers. No adapter has been run against a live Anthropic, OpenAI, Gemini, Azure, OpenRouter or any other real service, so wire details such as Azure URL shapes, Gemini schema acceptance, Ollama tool-call streaming quirks and the catalog's default model ids and brand marks may need adjusting on first real use. The self-hosted example above has not been run on a real GPU node. Prompt caching is requested for Anthropic but cache hits were not observed.
## Troubleshooting
### AI assistant page not showing or returns 404
Likely cause: The `APP_EXPERIMENTAL=ai-chat` flag is not set, or the control plane was not restarted after setting it.
How to confirm: Check the flag: `ps aux | grep levelrail | grep -i experimental` shows environment variables. Or try the API: `curl https:///api/v1/ai/sessions` should return a list (not 404).
Fix: Set `APP_EXPERIMENTAL=ai-chat` on the control plane and restart. Wait 10 seconds for the restart, then reload the dashboard page.
Still stuck: Check control plane logs for startup errors: `levelrail logs --since 1m | grep -i "ai\|experimental"`.
### "No usable AI provider is configured" when starting chat
Likely cause: No provider key is saved, or the active provider or model is not chosen.
How to confirm: `levelrail-cli settings ai-assistant get` shows the active provider and whether it is configured. `levelrail-cli settings ai-assistant providers list` shows which providers have a key.
Fix: `levelrail-cli settings ai-assistant providers set --api-key-env VAR`, then `levelrail-cli settings ai-assistant use `.
Still stuck: `levelrail-cli settings ai-assistant providers test ` re-checks the saved key and reports the status.
### Chat shows an error with a suggested fix
Likely cause: The provider rejected the call. The error shown names the class (see [Error meanings](#error-meanings)) and a fix.
How to confirm: `levelrail-cli ai chat "hello" --json` prints the `error` event with its `code`. `levelrail-cli settings ai-assistant usage` shows error counts per provider and model.
Fix: Follow the fix in the error. For `invalid_key` or `quota_exhausted`, check the key and credit at the provider; for `model_not_found`, pick a model from `providers models `.
Still stuck: Add a fallback provider so a transient provider problem does not stop the chat.
### Pending confirmation never resolves or takes too long
Likely cause: The confirmation request timed out, or the network connection was interrupted.
How to confirm: Check the session state: `levelrail-cli ai sessions get --json | jq .state` should show "pending\_confirmation". Look for a recent approval/rejection: `levelrail-cli ai sessions get --json | jq .confirmations[-1]`.
Fix: Either approve or reject the pending action: `levelrail-cli ai sessions resolve --approve` or `--reject`. If the session appears stuck, create a new session and chat again: `levelrail-cli ai chat "what happened?" --session `.
Still stuck: Check control plane logs for AI session errors: `levelrail logs --since 30m ai`.
### Chat works but tool results are wrong or missing
Likely cause: The AI assistant does not have permission to read the resource, or the tool call failed and the model retried with a workaround.
How to confirm: Check the session transcript: `levelrail-cli ai sessions get --json | jq '.messages[] | select(.role == "assistant") | .content'` shows tool calls and results. Look for "tool\_error" or "permission\_denied" in the results.
Fix: Verify the token you are using has `root` ability (required for all AI assistant routes and tools). If the tool call failed (for example, "app not found"), the input is wrong; try the query again or ask the model to try a different search.
Still stuck: See [AI assistant integration](/ai-assistant#untrusted-text-and-the-assistants-confirmation-gate) for the tool permissions model.
## See also
* [ai-assistant.md](/ai-assistant): `levelrail-mcp`, the standalone MCP server for an external AI client, and the confirmation/untrusted-text mechanism this chat shares with it.
* [experimental-features.md](/experimental-features): the `APP_EXPERIMENTAL` switch and what "off" means everywhere.
* [identity-and-access.md](/identity-and-access): API tokens and abilities (`ai` CLI commands and `/api/v1/ai/...` routes need `root`).
* [cli-reference.md](/cli-reference): full flag reference for `ai chat` and `ai sessions`.
# AI control
Source: https://docs.levelrail.com/ai-control
Decide what AI agents and agent tokens may do on your instance, from off to admin, enforced on the server for every request.
AI control is a switch in Settings that decides what agents may do on this instance. The control plane enforces it on every request, so it does not depend on the agent, the MCP client or the token's own abilities. The page is always visible because it is the guardrail.
## What counts as an agent
An agent is an API token that has an agent name, including the short-lived token the in-app assistant mints for each chat turn (agent name `ai-assistant`). That turn token is owned by the person who sent the message and carries only their abilities and IAM policies, so the mode below narrows what the assistant may do on their behalf, never widens it. Agent tokens can only be created by a signed-in person, never by another token.
AI control does not govern:
* people signed in with a session,
* API tokens that have no agent name (they are limited only by their abilities and IAM policies).
So give every automation or AI client token an agent name when you create it. Otherwise this switch cannot see it.
## Modes
| Mode | What an agent may do |
| - | - |
| `off` | Nothing. Every request answers 403 "agent access is disabled by an administrator". |
| `observe` | The `read` ability only, in any environment. No secrets, no changes. |
| `operate` | `read`, `read:sensitive`, `write` and `deploy`. Never `root`, never `write:sensitive` (secrets, backups, instance settings). On an app or database, only if its environment kind is allowed. |
| `admin` | Any ability the token holds, with the same environment limit. Needs `APP_EXPERIMENTAL=ai-control`. Without the flag it behaves as `operate`. |
In `operate` and `admin`:
* An agent can never approve a deploy approval, a pipeline approval or a preview approval. A person must.
* An agent can never change AI control settings or revoke tokens.
* Moving or deploying into a protected environment still creates an approval that a person decides.
The environment limit is a list of kinds: `dev`, `test`, `uat`, `production`, `preview`, `custom`. The default allowed list is `dev`, `test`, `uat` and `preview`. A resource with no environment counts as kind `custom`, which is not allowed by default.
## Set the mode
In the dashboard: Settings, AI control. Pick a mode and the allowed kinds. **Pause all agents** sets the mode to `off` in one click.
From the CLI:
```bash theme={"dark"}
levelrail-cli ai-control status
levelrail-cli ai-control set --mode operate --env-kinds dev,test,uat,preview
```
`--env-kinds` is optional: leave it out and the current list is kept. `--mode` is required.
To let agents act on resources with no environment, add `custom`:
```bash theme={"dark"}
levelrail-cli ai-control set --mode operate --env-kinds dev,test,uat,preview,custom
```
## Revoke agent tokens
```bash theme={"dark"}
levelrail-cli ai-control revoke-agents --yes
```
This revokes every token that has an agent name and reports how many. The in-app assistant's per-turn tokens are skipped: each one is already revoked when its turn ends. Set the mode to `off` to stop the assistant instead.
## Defaults and upgrades
* A fresh instance starts in `off`. The in-app assistant needs a mode set before it works.
* An instance that already had live agent tokens when it was upgraded starts in `operate`, so existing agents keep working.
## Audit
Every request an agent makes that this switch refuses is written to the audit log with the agent name and the reason. Changes to the setting and token revocations are audited too. Requests from the in-app assistant also record `on_behalf_of`, the person whose chat turn made them. To see one agent:
```bash theme={"dark"}
levelrail-cli audit-log --agent my-agent
```
The AI control page also links to the audit log filtered by agent.
## MCP
The MCP server reads the mode each time a client asks for its tool list, and hides what the mode would refuse, but only when its own token is an agent: `off` shows only `ai_control_status`, `observe` shows read tools, `operate` and `admin` show everything. A token with no agent name is not governed by this switch, so its tool list is never trimmed. This only tidies the list. The control plane enforces the mode whatever the client does.
## Error messages
| Message | Meaning |
| - | - |
| `agent access is disabled by an administrator` | The mode is `off`. |
| `agent access is limited by the current AI control mode` | The mode does not allow this ability, for example a write in `observe` or `root` in `operate`. |
| `agent access is not allowed in this environment` | The resource's environment kind is not in the allowed list. |
| `agents cannot approve deploys, a human must` | Approvals always need a person. |
| `agents cannot change AI control settings` | Only a person with `root` can change the switch. |
## API
| Method | Path | Ability |
| - | - | - |
| `GET` | `/api/v1/settings/ai-control` | `read` |
| `PUT` | `/api/v1/settings/ai-control` | `root` |
| `POST` | `/api/v1/settings/ai-control/revoke-agent-tokens` | `root` |
`GET` also reports `caller_is_agent`, whether the token making the request is an agent. `PUT` takes `{"mode": "...", "allowed_env_kinds": [...]}` and answers 400 for `admin` unless the `ai-control` flag is on.
## Next steps
Scope an API token for an MCP client or an AI agent.
Build an agent that deploys and monitors your apps.
# AI models
Source: https://docs.levelrail.com/ai-models
Run Ollama, vLLM or llama.cpp models on NVIDIA GPU nodes, or small GGUF models on CPU with hard limits, as a first-class resource with an OpenAI-compatible endpoint protected by an API key.
A model is a first-class resource: an inference engine running as a container on a GPU node, a persistent volume holding the downloaded weights, and an OpenAI-compatible endpoint protected by an API key. You can manage it from the dashboard (**AI models**), the CLI (`levelrail-cli models`), the REST API (`/api/v1/models`) and the MCP server.
Version 1 supports NVIDIA GPUs on Linux nodes (Ubuntu is the documented path). AMD, Intel and Apple GPUs are not supported.
## GPU nodes
Every node reports its NVIDIA GPUs to the control plane: driver version, per-GPU VRAM total and used, utilization, and whether Docker has the `nvidia` container runtime registered. Detection runs `nvidia-smi` on the host and asks the Docker Engine API for its runtimes. The control plane checks its own host every minute (`APP_GPU_COLLECT_INTERVAL`). Agents report when they connect and every minute after.
The **AI models** page shows a card per GPU node with VRAM used and total. `levelrail-cli models gpus` prints the same data.
### Prerequisites on the node (Ubuntu)
1. Install the NVIDIA driver: `sudo ubuntu-drivers install`, then reboot. `nvidia-smi` must work.
2. Add NVIDIA's apt repository for the container toolkit, then install and register it:
```sh theme={"dark"}
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```
When a node has a GPU but Docker has no `nvidia` runtime, the dashboard card shows "nvidia runtime missing" with this fix, `levelrail-cli doctor` and the [Status page](/status-page) raise a warning, and models placed there stay in the `GPURuntimeMissing` state until it is fixed. Run `levelrail-cli models restart ` afterwards.
If the agent runs inside a container, `nvidia-smi` must be reachable from it, or run the agent directly on the host.
## GPU attach: CDI and legacy
Containers get GPUs one of two ways. Levelrail prefers CDI (the Container Device Interface) when the Docker daemon on the node lists NVIDIA CDI devices (`nvidia.com/gpu=0`, `nvidia.com/gpu=all`), and otherwise uses the legacy `nvidia` runtime with a device request. The choice is made per container at create time on the node that runs it, so mixed fleets work. A node with CDI devices needs no `nvidia` runtime registered and is treated as usable.
To use CDI, generate a spec on the node and make sure Docker discovers it (Docker 28.3 or newer does by default; older Docker needs `{"features": {"cdi": true}}` in `/etc/docker/daemon.json`):
```sh theme={"dark"}
sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo systemctl restart docker
docker info | grep -A3 "Discovered Devices"
```
A request that CDI cannot serve exactly (for example more GPUs than the spec lists) falls back to the legacy runtime instead of failing. With a count, CDI attaches the first N GPUs by index. Set `APP_GPU_ATTACH=legacy` on a node to never use CDI (the default is `auto`). Regenerate the spec after a driver update or GPU change.
`levelrail-cli doctor` (and the Doctor page) checks the control plane host in detail: the driver, per-GPU memory (a warning above `APP_GPU_DOCTOR_VRAM_WARN_PERCENT`, default 90), the NVIDIA container toolkit version, whether a CDI spec exists and Docker lists its devices, whether the `nvidia` runtime is registered, and the resulting attach mode, with the exact fix command for each problem. Remote nodes report only driver, runtime or CDI availability and per-GPU memory; the toolkit and spec files of a remote node are not inspected. None of this was exercised against a real GPU: the detection and the CDI request are covered with fakes.
## Deploying a model
```sh theme={"dark"}
levelrail-cli models deploy --name chat --engine ollama --model llama3.1:8b
levelrail-cli models deploy --name llama --engine vllm --model meta-llama/Llama-3.1-8B-Instruct \
--gpus 2 --context 8192 --hf-token-from-env --node --domain llm.example.com
```
| Option | Meaning |
| - | - |
| `--engine` | `ollama`, `vllm` or `llamacpp`. |
| `--model` | An Ollama tag (`llama3.1:8b`), a HuggingFace repo (`org/name`) for vLLM, or a GGUF repo for llama.cpp, optionally with a quant suffix (`org/name-GGUF:Q4_K_M`). |
| `--node` | Node ID. Default is the control plane's own host. |
| `--gpus`, `--gpu-devices` | A count, `all`, or specific GPU indexes or UUIDs. |
| `--context` | Maximum context length in tokens. |
| `--quantization` | vLLM only (for example `awq`). Ollama encodes it in the tag, llama.cpp in the `:quant` suffix. |
| `--domain` | Hostname for the endpoint. Without one, a zero-config hostname is used when `APP_PUBLIC_HOST` is a public IP. |
| `--hf-token-from-env` | Reads `HF_TOKEN` from your environment for gated models. |
The engine image can be overridden per engine with `APP_MODEL_IMAGE_OLLAMA`, `APP_MODEL_IMAGE_VLLM` and `APP_MODEL_IMAGE_LLAMACPP`.
### HuggingFace token
The token is stored through the envelope-encryption path (`internal/secrets`) under the model's own namespace and injected into the engine container at create time. It is never returned by the API and never logged. It requires a master key (`APP_MASTER_KEY`). Rotate it with `PUT /api/v1/models/{name}/hf-token`, which restarts the model.
## Hugging Face preflight
Before deploying a Hugging Face repository, check it:
```sh theme={"dark"}
levelrail-cli models preflight bartowski/Llama-3.2-3B-Instruct-GGUF --engine llamacpp --node
HF_TOKEN=hf_... levelrail-cli models preflight meta-llama/Llama-3.1-8B-Instruct --engine vllm --hf-token-from-env
```
The check queries the public Hub API (through the same outbound guard as webhooks, with a timeout and a response size cap) and reports:
* whether the repository exists, and whether it is gated. A gated repository says what to do next: accept the license on huggingface.co and provide an access token.
* the license, total download size and per-file sizes.
* the GGUF quantizations it contains, with sizes, and a recommended one for the target node.
* whether the node has enough free disk for the download, and which engine can load the weights (GGUF for llama.cpp and Ollama, safetensors for vLLM).
In the dashboard the same panel appears under the model field of the deploy dialog and updates as you type. Picking a quantization there fills in the `:quant` suffix.
Fit figures are estimates. A quantization is judged against the node's free VRAM as its file size plus a flat overhead for the KV cache and buffers. Real usage depends on context length, batch size, engine and driver. Fit is shown only when the node has reported its GPU, and disk only when it has reported its disk; otherwise the answer is "unknown", never a guess. System RAM is not considered.
The token you supply is used for that one request. It is not stored or logged, and it is never part of a cache key.
| Variable | Default | Meaning |
| - | - | - |
| `APP_HF_CACHE_TTL` | `5m` | How long Hub answers are reused. `0` disables the cache. |
| `APP_HF_TIMEOUT` | `10s` | Per-request timeout to the Hub. |
| `APP_HF_MAX_RESPONSE_BYTES` | `16777216` | Cap on the repository listing. |
| `APP_HF_MAX_LISTED_FILES` | `100` | Files returned per repository (totals always cover all files). |
| `APP_HF_BASE_URL` | `https://huggingface.co` | Hub base URL (a mirror). |
| `APP_MODEL_FIT_OVERHEAD_PERCENT` | `20` | Added to weight size for KV cache and buffers. |
| `APP_MODEL_FIT_PERCENT` | `90` | Share of free VRAM a model may use to count as "fits". Above it, up to 100%, is "tight". |
| `APP_MODEL_DISK_HEADROOM_PERCENT` | `10` | Free disk required beyond the download. |
The disk check measures the filesystem holding Docker's volumes on the control plane host. If that path is not visible from the control plane, or the node is remote, disk is reported as unknown.
The API is `POST /api/v1/models/preflight` (`repo`, optional `engine`, `quant`, `file`, `gpu_count`, `node_id`, `hf_token`) and the MCP tool is `preflight_model`. Hub problems (missing, gated, rate limited, unreachable) come back as a `status` in a normal response so a client can always show the next step.
## VRAM fit check
Before you deploy, and on the model page, Levelrail estimates whether a model fits in each GPU node's free VRAM and rates every node: `fits`, `tight`, `wont_fit` or `unknown`. The pill has an info tip with the arithmetic, for example `weights 4.7 GiB + KV 1.2 GiB + overhead 0.8 GiB = 6.7 GiB of 8.0 GiB free`.
This is an estimate, not a guarantee. It is built from these pieces:
* Weights: Exact when the Hugging Face file sizes are known (pass `weights_bytes` from a preflight), otherwise estimated from the parameter count in the model name (`8b`, `8x7b`) and the quantization (a tag suffix such as `q4_K_M`, a `:quant` suffix, or vLLM's `--quantization`). A name with no size, such as `mistral`, is `unknown`, never guessed. Ollama and llama.cpp default to about 4.85 bits per weight, vLLM to 16 (bf16).
* KV cache: Approximated from the parameter count and the context length. With no context length set, `APP_MODEL_FIT_DEFAULT_CONTEXT` tokens (default 8192) are assumed and the tip says so.
* Overhead: A flat `APP_MODEL_FIT_GPU_OVERHEAD_MIB` per GPU (default 512), plus `APP_MODEL_FIT_VLLM_EXTRA_MIB` (default 1024) for vLLM.
* Free VRAM: From the node's GPU report (refreshed every minute). VRAM estimated for other models placed on the node that are not loaded yet is subtracted, so two deploys in the same minute do not both see the same free memory. On the model page, a loaded model's own VRAM is added back.
* Verdict: `fits` when the total is at most `APP_MODEL_FIT_PERCENT` of free VRAM (default 90), `tight` up to 100%, otherwise `wont_fit`. vLLM also needs each GPU to have `APP_MODEL_VLLM_GPU_UTILIZATION` (default 0.9) of its memory free, because it pre-allocates that much.
Nodes are ranked best first, with non-schedulable nodes last. The check is advisory: it never blocks a deploy. `APP_MODEL_FIT_KV_KIB_PER_TOKEN` overrides the KV heuristic if your model family differs.
* Dashboard: The deploy dialog shows a verdict per node and highlights the selected one. The model page Overview shows the same for the deployed model, with its current node marked.
* CLI: `levelrail-cli models fit --engine ollama --model llama3.1:8b --context 8192`, or `levelrail-cli models fit --name chat` for a deployed model.
* API: `POST /api/v1/models/fit` and `GET /api/v1/models/{name}/fit` (read ability).
* MCP: `check_model_fit` (read-only).
No GPU was involved in testing this; the numbers come from the GPU facts nodes report and have not been compared against real engine memory use.
## Model cache
Each model keeps its downloaded weights in its own Docker volume. Deleting a model keeps the volume, so those volumes are what accumulates.
```sh theme={"dark"}
levelrail-cli models cache list
levelrail-cli models cache prune --dry-run
levelrail-cli models cache prune
```
The list shows each volume with its size, its owning model, when it was last used, and whether it is safe to prune. Last use comes from gateway traffic when the model has any, otherwise from the model's last change (or, for a volume no model owns, the volume's creation time). Two models that point at the same weights are flagged and counted once in the unique total.
Prune removes only volumes that no model owns, that no container mounts, and that have been unused for `APP_MODEL_CACHE_UNUSED_DAYS` days (default 30). Weights of any deployed model, running or not, are never removed. The server rechecks each volume right before removing it. In the dashboard, the Model cache section shows a dry run in a confirmation dialog first. Pruning needs the root ability.
Only the control plane's own host is inspected today; remote nodes are listed as not yet supported. The MCP tool `list_model_cache` is read-only; the API is `GET /api/v1/model-cache` and `POST /api/v1/model-cache/prune`.
## CPU models
A small model can run on the server's CPU, with no GPU. It is the same resource as a GPU model (endpoint, keys, residency, usage), with `compute: cpu`, hard resource limits, a curated list and a download that Levelrail performs itself. Use it for short tasks on a modest server, not for chat at GPU speed.
```sh theme={"dark"}
levelrail-cli models catalog
levelrail-cli models fit --compute cpu --catalog granite-4.1-3b
levelrail-cli models deploy --compute cpu --catalog granite-4.1-3b --cpus 2 --memory-mib 4096
levelrail-cli models benchmark granite-4-1-3b
```
In the dashboard, **Deploy model** has a **Run on CPU** option: the curated models as cards (size, memory, licence, tier, estimated speed), CPU and memory sliders with the verdict updating as you move them, a plain warning of what it will use, a download progress bar, and a **Benchmark** button on the model page.
### What it runs
llama.cpp's official CPU server image (`ghcr.io/ggml-org/llama.cpp:server`). Override it with `APP_MODEL_IMAGE_LLAMACPP_CPU` (the GPU image stays `APP_MODEL_IMAGE_LLAMACPP`). Only llama.cpp supports CPU mode today; Ollama and vLLM models stay GPU only. The command has no `-ngl`, and these flags come from the limits:
| Flag | Value |
| - | - |
| `-t`, `-tb` | the whole CPUs of the cap (at least 1) |
| `-np` | `1`: one request at a time, so memory use stays predictable |
| `-c` | the context size of the resource (default `APP_MODEL_CPU_DEFAULT_CONTEXT`, 4096) |
| `-cram` | `APP_MODEL_CPU_CACHE_PERCENT` (5) of the memory cap, between 64 MiB and `APP_MODEL_CPU_CACHE_MAX_MIB` (512). llama.cpp's own default is 8192 MiB, which would eat a small server |
| `--no-mmap` | only when the cap leaves less than `APP_MODEL_CPU_MMAP_HEADROOM_MIB` (1024) beyond the weights, or when `APP_MODEL_CPU_MMAP=off` |
Memory mapping is the default: the kernel can drop weight pages when the whole host is short, and loading needs no second copy. With a cap barely above the weights, mapped pages would thrash the disk and slow every app sharing it, so the weights are read into memory instead and a too-small cap shows up as a clear out-of-memory kill. `APP_MODEL_CPU_MMAP=on` or `off` forces either.
### Limits
Every CPU model container is created with hard limits, so it cannot starve the apps next to it:
| Limit | Value |
| - | - |
| CPU | `NanoCPUs` from the cpu cap |
| Memory | the memory cap, with swap equal to memory (no swap) |
| CPU weight | `APP_MODEL_CPU_SHARES` (128, against Docker's 1024), so apps win under contention |
| OOM score | `APP_MODEL_CPU_OOM_SCORE_ADJ` (800): the kernel kills the model before app containers |
| Processes | `APP_MODEL_CPU_PIDS_LIMIT` (256) |
The caps come from the resource. Left empty they default to the smaller of 2 CPUs and half the node's CPUs (`APP_MODEL_CPU_DEFAULT_CPUS` overrides the 2), and `APP_MODEL_CPU_DEFAULT_MEMORY_PERCENT` (25) of the node's RAM. A request is refused above `APP_MODEL_CPU_MAX_CPU_PERCENT` (75) of the CPUs or `APP_MODEL_CPU_MAX_MEMORY_PERCENT` (60) of the RAM, or when the cap would leave less than `APP_MODEL_CPU_FREE_RAM_FLOOR_MIB` (1024) of the node's RAM. Only the control plane's own host reports its CPUs and RAM; for a remote node you must pass both caps. The CPU weight, OOM score and pids limit are applied on the local node only; they are not carried over the agent transport yet, the CPU and memory caps are.
### Preflight
`levelrail-cli models fit --compute cpu`, `POST /api/v1/models/fit` with `compute: cpu`, and the live panel in the deploy dialog return:
* Memory: weights plus KV cache at the chosen context plus overhead, against the cap.
* Disk: the weights download, with a free disk check.
* Speed: a range of tokens per second, always labelled an estimate. Generation is limited by memory bandwidth, so the range scales with the CPUs you allow (`APP_MODEL_CPU_BW_GBS_PER_CPU`, `APP_MODEL_CPU_BW_GBS_MAX`).
* A verdict, with the reason: `fits_comfortably` (at most `APP_MODEL_CPU_COMFORT_PERCENT` of the cap, 85), `tight`, `wont_fit` or `unknown` (the size cannot be read, never guessed), and a smaller context or model to try.
Deploying a model that will not fit is refused. A tight one is allowed with the warning.
### Curated list
The list is data (`internal/models/catalog_local.json`, embedded in the binary), not code. Only Apache-2.0 and MIT models are listed; the loader rejects any other licence. Sizes, licences, repositories and sha256 digests were read from the Hugging Face API on the date shown in the catalog. Levelrail never ships or mirrors weights: your server downloads them from Hugging Face when you deploy.
| Id | Tier | Download | Licence |
| - | - | - | - |
| `granite-4.0-1b` | light | 1.0 GB | Apache-2.0 |
| `qwen3.5-2b` | light | 1.3 GB | Apache-2.0 |
| `nomic-embed-text-v1.5` (embeddings) | light | 0.1 GB | Apache-2.0 |
| `granite-4.1-3b` | default | 2.1 GB | Apache-2.0 |
| `qwen3.5-4b` | plus | 2.7 GB | Apache-2.0 |
| `gemma-4-e4b` | plus | 5.0 GB | Apache-2.0 |
The list is provisional: the entries were checked against the Hub, not run on a CPU server here. Qwen3.5 thinks before answering by default, which is slow on CPU; Granite 4.1 does not.
Any other repository goes through the custom repository field (`--model org/name-GGUF:Q4_K_M`, plus `--weights-file` if the suffix does not pick one file). It shows its licence as read from the Hub with a notice that Levelrail did not review it, and the deploy is refused until you acknowledge it (`--accept-license`, or the checkbox). Gemma 3 and older, and Llama licences, are reachable only this way.
### Download
For a CPU model on the control plane's own host, Levelrail downloads the GGUF file itself into `$APP_DATA_DIR/model-weights//` (`APP_MODEL_WEIGHTS_DIR` overrides it) and mounts it read-only into the engine. This assumes the control plane runs as a process on the host, so the directory is visible to Docker; a control plane running inside a container needs `APP_MODEL_WEIGHTS_DIR` pointing at a path that is the same on the host. The weights are never baked into an image.
* Only `huggingface.co`, `hf.co` and their subdomains are allowed, over HTTPS, including every redirect (Hugging Face serves files from a CDN). Internal addresses are refused by the outbound guard.
* Interrupted downloads resume with an HTTP `Range` request. A server that ignores `Range` restarts from zero.
* The sha256 is verified: from the catalog, else from the digest the Hub reports. A mismatch discards the file and fails with `checksum_mismatch`.
* Limits: `APP_MODEL_DOWNLOAD_MAX_BYTES` (16 GiB), free disk plus `APP_MODEL_DISK_HEADROOM_PERCENT`.
* Progress (bytes done and total) is in the resource status: the model page, `models get` and `GET /api/v1/models/{name}` (`download`).
Failures are stated plainly in the status: `gated` (accept the licence on huggingface.co and set a token), `not_found`, `disk_full` (the partial file is kept), `interrupted` (resumes), `checksum_mismatch`, `too_large`, `host_not_allowed`. If the download finishes but the engine fails to start, the weights stay on disk and the next attempt reuses them. Deleting the model keeps the weights, like the GPU volume. On a remote node the engine pulls the weights itself and there is no progress; Ollama progress is not exposed either.
### Benchmark
`POST /api/v1/models/{name}/benchmark`, `levelrail-cli models benchmark ` or the **Benchmark** button runs three short fixed prompts through the running server and stores the last result: generation and prompt tokens per second, time to first token, and peak memory read from the container. It has a hard time cap (`APP_MODEL_BENCH_TIMEOUT_SECONDS`, 120), allows one run at a time, is rate limited per model (`APP_MODEL_BENCH_MIN_INTERVAL_SECONDS`, 300; the API answers `429` with `Retry-After`), never wakes a stopped model, and never runs on its own. The dashboard shows the measured speed next to the estimate.
### Idle unload
Use `--residency on_demand` (the dashboard default for CPU models). After the idle time (`--idle-ttl`, `APP_MODEL_IDLE_TTL`) the engine stops and frees its memory and CPU; weights stay on disk, so the next request only waits for the model to load again.
### Estimated and unverified
Everything here was tested against fakes and unit tests: flag and limit generation, verdicts, the downloader against a local HTTP server, the controller against a fake Docker client. No container was started and no model was run. The tokens per second range, the memory needed for each model and the quality of each model on a CPU are estimates. The model names postdate what was verified beyond the Hub listing. Run the benchmark on your server before relying on any number.
## Lifecycle and status
A model has one `Ready` condition whose reason tells you where it is:
| Reason | Meaning |
| - | - |
| `Starting` | The container is being created or started. |
| `Downloading` | The weights are downloading. Ollama reports percent and bytes; vLLM and llama.cpp download on start, follow `levelrail-cli models logs --follow`. |
| `Loading` | The model is being loaded into VRAM (or memory, for a CPU model). |
| `DownloadFailed` | A CPU model's weights download failed. The message says why; retryable failures resume on their own. |
| `CPULimitsMissing` | A CPU model has no cpu or memory cap. Redeploy it with limits. |
| `ModelLoaded` | The model is loaded and serving. This is the readiness signal. |
| `Idle` | An on-demand model whose engine was stopped after its idle time. It starts on the first request (see On-demand residency). |
| `WakingUp` | An on-demand model's engine is starting or loading after a wake. |
| `WaitingForGPU` | An on-demand model would fit the GPUs but other workloads hold too much VRAM right now; it starts when they free it. |
| `NoGPUOnNode`, `GPURuntimeMissing`, `InsufficientGPUs` | The node cannot serve the request. See GPU nodes above. |
| `HFTokenUnavailable` | A token is set but cannot be decrypted (no master key). |
| `EndpointUnreachable` | The node is remote and not on the WireGuard mesh, so the control plane cannot reach the engine. |
| `DownloadFailed` | The engine reported a download error (retried after 30 seconds). |
The downloaded weights live in a persistent Docker volume, so a restart or redeploy does not download again. Deleting a model removes its container but keeps the volume.
## The model page
Each model has its own page at `/models/` (click its name in the list). Tabs:
* Overview: Status, engine, node, GPU, context length, quantization, the endpoint URL with a copy button, and a ready-made `curl` example.
* Keys: the named keys with their limits, expiry and rotation (see Virtual keys below).
* Usage: Requests, tokens, errors and first-byte latency per key.
* Logs: The engine's live log tail.
The tab is kept in the URL (`?tab=keys`), so a link opens straight to it. From the CLI, `levelrail-cli models get ` prints the same overview facts.
## Engine metrics
The Overview tab shows the inference engine's own metrics with a health summary. The control plane scrapes each running engine every `APP_MODEL_ENGINE_METRICS_INTERVAL` (default 15s) over the same path its readiness probe uses, and stores the readings in the telemetry database under `model:`.
| Metric | vLLM | llama.cpp | Ollama |
| - | - | - | - |
| KV cache usage | yes | yes | not available |
| Queued and running requests | yes | yes | not available |
| Prefix cache hit rate | yes | not available | not available |
| Tokens per second | yes | yes | not available |
| Time to first token | yes | not available | not available |
| VRAM in use, share running on CPU | not available | not available | yes (from `/api/ps`) |
Rates (tokens per second, prefix hit rate, time to first token) are computed between two scrapes, so they appear one interval after the model starts and skip an engine restart. llama.cpp serves `/metrics` only when started with `--metrics`, which new containers now get; a model deployed earlier needs `levelrail-cli models restart `. Ollama exposes no Prometheus metrics, so the page says "Not available for ollama" instead of guessing; per-key tokens and first-byte latency are on the Usage tab.
The health pill warns when the KV cache is at or above `APP_MODEL_KV_WARN_PERCENT` (default 90), when more than `APP_MODEL_QUEUE_WARN` requests are queued (default 4), or when an Ollama model runs partly on CPU.
* CLI: `levelrail-cli models metrics --since 6h`.
* API: `GET /api/v1/models/{name}/engine-metrics?since=1h` (read ability).
* MCP: `get_model_engine_metrics` (read-only).
The scrape is done by the control plane, not by a node agent, so a remote node's engine must be reachable over the mesh (the same requirement as serving it). Numbers depend on the engine version's metric names; vLLM's older `gpu_cache_usage_perc` and current `kv_cache_usage_perc` are both read.
## On-demand residency
By default a model is always loaded. On a shared GPU you can make it on demand: after an idle period the engine container is stopped, freeing its VRAM, and the first request starts it again.
```sh theme={"dark"}
levelrail-cli models deploy --name chat --engine ollama --model llama3.1:8b --residency on_demand --idle-ttl 20m
levelrail-cli models residency chat --mode on_demand --idle-ttl 30m
levelrail-cli models wake chat # start now
levelrail-cli models sleep chat # stop now
```
How it behaves:
* Idle: Every authenticated gateway request marks the model used (a long streaming response keeps refreshing it). When nothing has used it for the idle TTL, the reconciler stops the container and the status reason becomes `Idle`. The TTL is per model (`--idle-ttl`), defaulting to `APP_MODEL_IDLE_TTL` (15m). It cannot be set below `APP_MODEL_MIN_IDLE_TTL` (1m), so a busy model is never judged idle between activity writes (`APP_MODEL_ACTIVITY_TOUCH_INTERVAL`, default 10s).
* Wake: The first authenticated request to an idle model asks the reconciler to start it and is held while the engine loads, for at most `APP_MODEL_WAKE_WAIT` (default 90s). If the engine is not serving by then, or the wait is 0, the client gets `503` with `Retry-After` (`APP_MODEL_WAKE_RETRY_AFTER`, default 15s) and error code `model_waking`; the wake keeps going, so a retry usually succeeds. Unauthenticated or invalid requests never wake a model.
* Ollama: Ollama's own keep-alive is set to forever so weights stay in VRAM; unloading it means stopping the container, which is what on-demand does.
* GPU contention: If a wake would fit an empty GPU set but other workloads hold too much VRAM (per the fit estimate above), the model reports `WaitingForGPU` and starts when memory frees. A model too large for the GPUs is not held back, since waiting would not help. The GPU snapshot refreshes every minute, so this can lag.
* State: The reconciler owns the asleep, waking and awake states. Stop and start are idempotent: a stop that succeeded but failed to record is repaired on the next pass, and a request that lands while the reconciler is deciding to stop wins (the row is re-read right before stopping).
* Dashboard: the model page Overview has a Residency card (mode, idle minutes, Wake now, Sleep now) and the deploy dialog has a Residency field. API: `PUT /api/v1/models/{name}/residency`, `POST /api/v1/models/{name}/wake`, `POST /api/v1/models/{name}/sleep` (write ability); the model resource carries `residency`, `idle_ttl_seconds`, `effective_idle_ttl_seconds`, `residency_state` and `last_active_at`. MCP: `get_model` returns the same fields and `deploy_model` accepts `residency` and `idle_ttl_seconds`.
Swap groups (several models sharing one GPU, only one loaded at a time) are not built; give each model on a shared GPU an idle TTL instead. Cold start includes the engine's load time (minutes for a large vLLM model), so on-demand suits models used in bursts, not latency-critical ones. No GPU was involved in testing this; the engine start, stop and wake paths are covered with fakes.
## The endpoint and API key
The model's hostname is routed through the built-in ingress to the control plane, whose gateway checks the API key and proxies to the engine. It speaks the OpenAI API:
```sh theme={"dark"}
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer lr-..." -H "Content-Type: application/json" \
-d '{"model": "llama3.1:8b", "messages": [{"role": "user", "content": "hello"}]}'
```
* The key is generated at deploy time and shown once. Only its SHA-256 hash is stored. Lost it? `levelrail-cli models rotate-key ` replaces the `default` key at once. For more than one key, see [Virtual keys](#virtual-keys-and-usage).
* Only an allowlist of OpenAI-compatible routes is served (see below). Engine admin APIs (Ollama's pull and delete, vLLM's runtime LoRA load and unload, llama.cpp's `/props` and `/lora-adapters`) are never exposed, even ones that live under `/v1/`.
* The engine's port is published on loopback (local node) or the WireGuard mesh address (remote node), never on a public interface.
### Served routes
Matching is exact and case sensitive: trailing slashes, `//`, `..`, backslashes and encoded slashes (`%2F`) are rejected. Anything else returns a `404` OpenAI-style error before the API key is checked, and a wrong method returns `405` with an `Allow` header.
| Route | Method | Ollama | vLLM | llama.cpp |
| - | - | - | - | - |
| `/v1/models`, `/v1/models/{id}` | GET | yes | yes | yes |
| `/v1/chat/completions` | POST | yes | yes | yes |
| `/v1/completions` | POST | yes | yes | yes |
| `/v1/embeddings` | POST | yes | yes | yes |
| `/v1/responses` | POST | yes | yes | yes |
| `/v1/audio/transcriptions`, `/v1/audio/translations` | POST | no | yes | no |
Not served on purpose: vLLM's `/v1/load_lora_adapter`, `/v1/unload_lora_adapter`, `/v1/chat/completions/batch` and the `/render` routes; llama.cpp's `/v1/chat/completions/control` and its Anthropic-style `/v1/messages`; stateful `GET` and cancel on `/v1/responses/{id}`.
### Limits
Limits are global (every model) and set with environment variables on the control plane. `0` or a negative value disables a limit. The current values show on `GET /api/v1/models/{name}` (`limits`) and in `levelrail-cli models get`.
| Variable | Default | Effect |
| - | - | - |
| `APP_MODEL_GATEWAY_MAX_BODY_BYTES` | `33554432` (32 MiB) | Larger request bodies get `413`. |
| `APP_MODEL_GATEWAY_MAX_N` | `16` | `n` or `best_of` above this gets `400`. |
| `APP_MODEL_GATEWAY_MAX_TOKENS` | `32768` | `max_tokens`, `max_completion_tokens` or `max_output_tokens` above this, or negative (unlimited on llama.cpp), gets `400`. A request that sets none is passed through; the engine's context length bounds it. |
| `APP_MODEL_GATEWAY_MAX_INFLIGHT` | `32` | Concurrent requests per model; more get `429` with `Retry-After`. Streams hold a slot until they end. |
| `APP_MODEL_GATEWAY_RETRY_AFTER` | `5s` | The `Retry-After` value. |
| `APP_MODEL_GATEWAY_DIAL_TIMEOUT` | `5s` | Connecting to the engine. |
| `APP_MODEL_GATEWAY_HEADER_TIMEOUT` | `5m` | Waiting for the engine's response headers. A non-streaming completion sends none until it finishes, so keep this above your longest generation. Exceeded: `504`. |
| `APP_MODEL_GATEWAY_IDLE_TIMEOUT` | `2m` | Longest gap between engine output on a response. It is a gap, not a total, so long streams that keep producing tokens are never cut. |
JSON bodies are read up to the size cap so `n` and the token limits can be checked; the body is then forwarded unchanged. Audio uploads are size-capped but not parsed. Request and response bodies are never logged: each request logs only method, model, status, duration and bytes.
## Virtual keys and usage
A model can have several named keys, so each client gets its own identity, limits and usage. The key made at deploy time is the key named `default`; existing models were migrated to it, and it keeps working unchanged.
```bash theme={"dark"}
levelrail-cli models keys create chat --name ci --rpm 60 --tpm 100000 --tpd 2000000 --max-parallel 4 \
--allow-paths /v1/chat/completions --expires-in 720h
levelrail-cli models keys list chat
levelrail-cli models keys rotate chat --grace 30m
levelrail-cli models keys revoke chat
levelrail-cli models usage chat --since 168h
```
In the dashboard, the gauge button on a model row opens the keys panel (create, rotate, revoke, last used, limits) and the usage card (requests, tokens, errors and time to first byte over time, plus a per key table).
* Only the SHA-256 of a key is stored. The key is shown once, with its first 8 characters kept as a prefix for identification. `last used` is updated on each flush interval.
* Rotation issues a replacement with the same name, limits and expiry. The old key keeps working for a grace window (`--grace`, default `APP_MODEL_KEY_ROTATION_GRACE`, `1h`; at most `APP_MODEL_KEY_MAX_GRACE`, `168h`; `0` retires it at once), then stops. Revoking stops a key immediately.
* Limits are per key. `rpm` and max parallel are enforced at the gateway: a request over either gets `429` with `Retry-After` and the same generic OpenAI-style error, so the response never says which limit tripped. `tpm` and `tpd` (tokens per day, rolling 24 hours) are soft: tokens are counted from responses after they finish, so the request that crosses the limit still completes and later ones get `429` until the window rolls over. An unknown, revoked or expired key always gets `401`.
* Allow lists: `allow_paths` are exact gateway paths (a trailing `/` allows everything below it) and must be routes the engine's allowlist already serves, so a key can never reach an engine admin route. `allow_models` are compared with the `model` field of the request body; a request without one is refused when the list is set.
* Each key records who created it (`created_by`). The MCP server can list keys (`list_model_keys`) and revoke one (`revoke_model_key`).
* At most `APP_MODEL_MAX_KEYS` (`50`) live keys per model.
### What is metered
Per key and model, per hour: requests, 2xx, 4xx and 5xx counts, requests refused with `429`, input and output tokens, response bytes, total duration and time to first byte (the first byte the engine sends, not the first token).
Tokens are read from the response's `usage` object: non-streaming responses, and streams opened with `stream_options.include_usage`. A stream without it, or a compressed response, is counted as a request only. Nothing is estimated, and the report says so (`usage_requests` shows how many requests carried usage). Prompts and bodies are never stored or logged; only the last `APP_MODEL_USAGE_SCAN_BYTES` of a JSON or event-stream body are held in memory to find `usage`.
Counts are aggregated in memory and written in bounded batches, so the request path never waits on the database. If the control plane stops between flushes, up to one interval of counts is lost.
| Variable | Default | Effect |
| - | - | - |
| `APP_MODEL_USAGE_FLUSH_INTERVAL` | `30s` | How often aggregates are written. |
| `APP_MODEL_USAGE_BATCH_SIZE` | `200` | Rows per write. |
| `APP_MODEL_USAGE_MAX_BUFFERED` | `10000` | Hourly aggregates held between flushes; more are dropped and logged. |
| `APP_MODEL_USAGE_RETENTION` | `720h` | Hourly rows older than this are deleted. `0` keeps them. |
| `APP_MODEL_USAGE_SCAN_BYTES` | `16384` | Tail of a response searched for `usage`. `0` turns token counting off. |
| `APP_MODEL_KEY_MAX_LIMIT` | `10000000` | Largest rpm, tpm or parallel value a key may be given. |
The control plane has no Prometheus endpoint yet, so usage is exposed through `GET /api/v1/models/{name}/usage?since=24h`, the CLI and the `get_model_usage` MCP tool.
## GPU apps
Any app can request a GPU with `resources.gpu` in `app.yaml` (see the [app spec reference](/app-spec-reference)). Such an app only runs on a node with a usable GPU: the reconciler reports `NoGPUOnNode` or `GPURuntimeMissing` instead of starting it, and moving it to a node without one is rejected.
## GPU scheduling
Placement counts GPUs, not just workloads. Each node has a ledger built from the desired state: every app with `resources.gpu` and every model on the node reserves GPUs.
| Request | Reserves |
| - | - |
| a count, `gpus: 2` | that many GPUs, anonymous |
| device IDs (index or UUID) | those exact devices; a second workload asking for the same device does not fit |
| `all` or no count | every GPU, so the node must be completely free |
A workload fits a node when the node reports a GPU, Docker has the nvidia runtime, and enough GPUs are free (total minus reserved, never below zero). A node that is exactly full fits nothing more. A workload's own reservation is ignored when re-checking it against its own node.
Where the ledger is consulted:
* Create: A new GPU app without `node_id` is auto-placed on the least loaded node that fits (the local host is the fallback), or refused with `409` and a per-node reason. An explicit `node_id` is checked the same way once the node has reported.
* Move: `PUT /api/v1/apps/{name}/node` rejects a target that lacks the runtime or the free GPUs.
* Drain: Each GPU app is placed on a node that fits, and GPUs it takes are reserved for the apps after it in the same drain. An app no node can host stays put and is reported under `blocked` (dashboard drain dialog, `levelrail-cli nodes drain`, API). Models are never moved, so they are always listed as blocked.
Docker does not isolate GPUs by count: two containers can share a device, so the ledger is scheduling accounting, not enforcement. A running app is never stopped because the ledger says the node is oversubscribed.
### Seeing reservations
* API: `GET /api/v1/gpus` adds `reserved_gpus`, `free_gpus` and `reservations` (`app:`, `model:`) per node. `GET /api/v1/nodes` and `GET /api/v1/nodes/{id}` carry a `gpu` summary.
* CLI: `levelrail-cli nodes list` has a GPU column (`free/total`), `levelrail-cli nodes get` prints a GPU block, `levelrail-cli models gpus` has a RESERVED column.
* Dashboard: The AI models page GPU cards and the node detail GPU card show reserved vs total GPUs and VRAM used vs total; the node list shows a `GPU free/total` badge.
* MCP: `list_gpu_nodes` returns the same fields.
### Attention and doctor
When a GPU app or model cannot run on its own node and no eligible GPU node has enough free GPUs, `levelrail-cli doctor` and the attention list (Status page, `levelrail-cli attention`, MCP `get_attention`) raise a warning `GPU app cannot be placed`. Fix it by freeing a GPU (stop or shrink another GPU workload), adding a GPU node, or installing the nvidia container toolkit on the node that has GPUs.
## Access control
Listing and reading models and GPUs needs the `read` ability. Deleting and restarting needs `write`. Listing keys and reading usage need `read`; revoking a key needs `write`. Deploying, creating or rotating a key and setting a HuggingFace token need `write:sensitive`. Model resources can be targeted by IAM policies as `model:`. The AI assistant asks for confirmation before any of the mutating model tools.
## Not in version 1
AMD and Apple GPUs, MIG partitioning, CPU mode for Ollama and vLLM, control plane downloads for remote nodes, automatic model-to-node scheduling (you pick the node; apps are spread, see GPU scheduling), moving a model between nodes, per-model limit overrides, swap groups, and a Prometheus endpoint for usage.
## GPU in Compose templates
A Compose file can request NVIDIA GPUs with the standard
`deploy.resources.reservations.devices` block. Levelrail reads `driver`
(only `nvidia` or unset), `count` (a number or `all`, unset means all),
`device_ids` and `capabilities: [gpu]`, and maps it onto `resources.gpu`.
The service then only starts on a node with a working NVIDIA runtime.
Catalogue entries that need a GPU carry a "Needs NVIDIA GPU" badge.
# All pages
Source: https://docs.levelrail.com/all-pages
Every Levelrail documentation page with a one-line summary, grouped by section.
The complete index of the Levelrail docs. For a self-hosted app gallery, see the [template gallery](/self-host).
## Get started
### Install and deploy
* [Levelrail documentation](/index): Self-hosted deployment platform. Push to git, get a running app with TLS, logs, metrics and rollback. One Go binary, no Kubernetes.
* [What you need before you start](/what-you-need): A plain-language checklist for installing Levelrail: a Linux server, a few ports, and optionally a domain. No Kubernetes, no prior DevOps experience.
* [Installing Levelrail](/installing): Requirements, then every supported way to get the Levelrail control plane running on Linux with install.sh, Docker, or source build.
* [Getting started](/getting-started): Install Levelrail on a Linux server, sign in, and deploy your first app in about ten minutes, from the dashboard or the CLI.
### Add to your setup
* [Domains and ingress](/domains-and-ingress): Automatic TLS, domain routing, WAF, rate limiting, and redirects, all embedded in the control plane, with no separate proxy to manage.
* [Connect an app to a managed Postgres database](/tutorials/connect-an-app-to-postgres): Create a managed Postgres database on your own server, connect an app with one command, and confirm the connection string reaches the container.
* [Deploy to your own server from GitHub Actions](/tutorials/deploy-from-github-actions): Build an image in GitHub Actions and deploy it to your own server with Levelrail on every push, failing the pipeline if the rollout fails. About 15 minutes.
* [Environment variables and secrets](/environment-and-secrets): Masking, reveal policies, and protection for app environment variables and secrets.
* [Back up Postgres to S3 and test the restore](/tutorials/back-up-postgres-to-s3): Send Postgres backups to S3, R2, or any S3-compatible bucket on a schedule, verify them, and prove the restore works on your own server.
* [Zero-downtime deploys with health checks](/tutorials/zero-downtime-deploys-with-health-checks): Add a readiness health check so a bad release fails before it takes over, read the structured failure, then fix it or roll back.
* [Debug an app with built-in logs and metrics](/tutorials/debug-an-app-with-logs-and-metrics): Find out why an app is failing or slow with Levelrail's built-in log search and metrics, with no Grafana or Loki to install.
* [Multi-node quickstart](/multi-node-quickstart): The short path to a second server, what has been verified on real infrastructure, and what to check before relying on the WireGuard mesh across hosts.
* [Self-host Vaultwarden on your own server](/tutorials/self-host-vaultwarden): Self-host Vaultwarden, a Bitwarden-compatible password manager, with one Levelrail command, then add a domain and back up its data.
* [Troubleshooting](/troubleshooting): Fixes for the most common problems when deploying, logging in, or running Levelrail.
### Guides
* [Guides](/guides/index): Task-focused Levelrail guides: compare it with Coolify, self-host Postgres with verified backups, and ship zero-downtime deploys on plain Docker.
* [Coolify vs Levelrail for self-hosters](/guides/coolify-vs-levelrail): A fair Coolify vs Levelrail comparison for self-hosters: how each manages servers, what each does well, what Levelrail lacks, and how to choose between them.
* [Self-host Postgres with verified backups](/guides/self-host-postgres-with-backups): Run managed Postgres on your own server and back it up to S3-compatible storage on a schedule, with a restore you have actually tested.
* [Zero-downtime deploys without Kubernetes](/guides/zero-downtime-deploys-without-kubernetes): Ship containers to your own servers with no downtime and a fast rollback, using readiness probes, pinned images and freeze windows.
### Announcements
* [Announcements](/announcements/index): News about Levelrail: new guides, the self-host template gallery, and platform updates.
* [Self-host template gallery and guides are live](/announcements/self-host-template-gallery-and-guides): Browse 330 one-click self-host templates and read new guides on backups, zero-downtime deploys and moving from Coolify.
## How-to guides
### Essentials
* [Troubleshooting](/troubleshooting): Fixes for the most common problems when deploying, logging in, or running Levelrail.
* [Email notifications](/email-notifications): Configure SMTP, AWS SES, or Resend as the backend for alert emails and password resets, and send a test email to confirm it works.
* [Attention center](/attention-center): How the dashboard surfaces things that wait on an operator, including CLI login approvals, expired logins, deploy approvals, expiring tokens, failed data copies, overdue backups and stalled certificate renewals.
### Deploying
* [Deploying and managing apps](/deploying-apps): Create, deploy, roll back, configure, promote, and operate an app on Levelrail, from the dashboard, CLI, or API.
* [Compose apps](/compose-apps): Run a Docker Compose file as one app with per-service status, start order, rollback to an earlier compose file, teardown, and installable templates.
* [New app wizard](/new-app-wizard): How the New app wizard reads a repository, pre-fills build, port, health path, env keys and databases with evidence, and deploys with one click.
* [Importing apps](/importing-apps): Import an app from a repo URL, a docker run command, an image, a compose file or a Dockerfile, review the deployment plan, then deploy it.
* [Deploy safety](/deploy-safety): What stands between a deploy being triggered and that exact content serving, a readiness gate with a stable window, a drain, pinned rollback targets, a post-deploy watch with automatic rollback, digest-truthful deploys, a stale-deploy guard and freeze windows.
* [Deploy failures](/deploy-failures): The structured failure object every failed or blocked deploy carries, and what each failure code means.
* [Scheduled deploys](/scheduled-deploys): Redeploy the latest commit on a branch on a recurring cron schedule, with freeze-window and restart-safe catch-up semantics.
* [Scheduled jobs](/scheduled-jobs): Run commands on a cron schedule against your app, in a fresh container with the app's env, secrets and volumes, with run history, logs, retries and alerts.
* [Image auto-update](/image-auto-update): Redeploy an image-based app automatically when its tag points at a new digest, with the same freeze and protection gates as a manual deploy.
* [Sleep when idle](/sleep-when-idle): Stop an app that has had no requests for a while so it uses no memory or CPU, and wake it on the next request.
* [Functions](/functions): Run an image as a function that sleeps when idle and answers the first request after a cold start, like a serverless function on your own server.
* [Canary deploys](/canary-deploys): Send a share of an app's traffic to a new image before promoting it, with one-command promote and abort.
* [Deployments page](/deployments-page): The cross-app Deployments page shows every deploy in one live list, with filters kept in the URL, a details drawer, redeploy and rollback actions, and keyboard shortcuts.
* [Deploy previews](/deploy-previews): Thumbnails of each deploy in three tiers, from free (page metadata, no browser) to a real screenshot from a short-lived browser container. What each tier costs, what it can leak, how it is cleaned up and every APP\_PREVIEW\_\* setting.
* [Deploy links and badge](/deploy-link): Put a "Deploy with" button in your README. A visitor enters their own server once and lands on its create-app form with your image, port, env variable names and health path prefilled, never submitted automatically.
* [Preview environments per pull request](/previews): Per pull request preview environments on your own servers, with their own domain and DNS record, safe defaults for data and secrets, a TTL and cap, idle sleep, and a PR comment that links back.
* [Chat-interactive deploy approvals](/chat-deploy-approvals): Approve or deny a protected-environment deploy directly from the Slack or Discord notification, without opening the dashboard.
* [Supply chain visibility](/supply-chain): A software bill of materials for each Dockerfile build, an optional vulnerability scan in a short-lived container and a gate that can keep a release from going live. What it costs, what it does not cover and every APP\_BUILD\_ATTEST, APP\_SCAN and APP\_SBOM setting.
* [Deploy status badge](/deploy-status-badge): An opt-in, per-app SVG badge showing an app's latest deploy status and when it happened, for embedding in your own project's README.
* [Dashboard screenshots](/screenshots): Regenerate real dashboard screenshots from a live control plane with metrics and logs.
* [Running Levelrail in Docker](/docker): Run Levelrail control plane and node agent as Docker containers.
* [Zero-downtime deploys with Docker: readiness-gated cutover and rollback](/zero-downtime-deploys): How Levelrail ships without downtime: blue-green, rolling or recreate strategies gated on real readiness probes and digest-pinned images.
* [Preview environments per pull request on your own servers](/preview-environments): Levelrail deploys a preview environment for each pull request on GitHub, GitLab, Bitbucket and Gitea and tears it down when the pull request closes.
### Networking and multi-node
* [Domains and ingress](/domains-and-ingress): Automatic TLS, domain routing, WAF, rate limiting, and redirects, all embedded in the control plane, with no separate proxy to manage.
* [Standalone domains](/standalone-domains): Proxy a domain to a port on this server, an address on your network, an external URL, another app's port or a container, with no app needed, and find the server address your DNS records must point at.
* [DNS zones and records](/dns): Manage DNS zones and records at Cloudflare or Route53, move a domain's name servers, verify delegation and propagation, and serve wildcard subdomains.
* [Domain traffic controls](/domain-traffic-controls): Per-domain traffic controls in the embedded ingress, covering header rules, path forwarders, country rules, response caching, redirects and raw ports, from the dashboard, CLI or API.
* [Verifying real ACME issuance against a live domain](/acme-verification-runbook): Verify real ACME certificate issuance against a live public domain with Let's Encrypt or another RFC 8555 CA, including what to check, common failures, and a recorded run.
* [Multi-node: adding and managing nodes](/multi-node): Enroll additional nodes, manage placement and health, migrate apps with their volumes, and configure the WireGuard mesh.
* [Multi-node quickstart](/multi-node-quickstart): The short path to a second server, what has been verified on real infrastructure, and what to check before relying on the WireGuard mesh across hosts.
* [Node provisioning](/node-provisioning): Create a server at Hetzner, DigitalOcean, AWS, Azure or GCP and enroll it as a node, with the credential each provider needs, how status polling works, cost, and known gaps.
* [Multi-cloud provisioning quickstart](/multi-cloud-provisioning): Pick a cloud provider and add a provisioned node in three commands, with the credential each provider needs. Auth modes, cost and known gaps are on the node provisioning page.
* [Network topology view](/network-topology): Read the Network page, how this server and nodes group into cards, how to turn the mesh on, and what the connection and reachability notes mean.
* [Project topology graph](/service-topology-graph): Read a project's topology diagram, what each node and edge represents, and exactly which stored field each edge is derived from.
* [Build node routing](/build-node-routing): Which node a deploy's build runs on, how to mark a node build-only, and what to do when a single node is under sustained load.
### Databases and storage
* [Managing databases](/managing-databases): Create and manage databases with backups, point-in-time restore, and lifecycle management.
* [Database upgrades](/database-upgrades): Schedule automatic database upgrades with security checks and automatic rollback on failure.
* [External databases](/external-databases): Connect a database that already runs somewhere else, or adopt a running container, without moving or changing any data, and point apps at it.
* [Connecting apps to databases](/connecting-apps-to-databases): Connect apps to multiple databases and view reachability badges.
* [Backup targets, registry credentials, and app volume backups](/backups-and-storage): Configure backup targets, registry credentials, and automated backup schedules with retention policies.
* [Prove your backups work](/prove-your-backups-work): Run restore drills to verify backups work before you need them.
* [Object storage](/object-storage): Connect S3-compatible buckets for log archive, build cache, and backups.
* [Network shares](/network-shares): Register and test network shares (NFS and CIFS) with the control plane.
* [Log explorer](/log-explorer): Search logs, save queries, group errors into signatures, and create alert rules.
* [Data export](/data-export): Send logs, the audit log and metrics to Loki, syslog, a webhook, a bucket, Prometheus or an OTLP collector, with bounded buffers and no effect on deploys.
* [Control plane backup and restore](/control-plane-backup): Back up and restore the control plane's database with automatic snapshots and offline restore.
* [Disaster recovery](/disaster-recovery): Encrypted off-box backups of the control plane database, master key escrow, offline restore and scheduled restore drills, with the threat model.
* [Upgrade safely](/upgrade-safely): Upgrade the control plane safely with pre-flight checks and automatic rollback.
* [Database access and network](/database-access): Manage database users, issue short-lived credentials, see who can reach a database, and control its network exposure, scope and TLS from the dashboard, CLI or API.
* [Database explorer and fleet overview](/database-explorer): How the database explorer lists tables, what its empty and error states mean, and what the dashboard shows for databases, reserved resources and storage.
* [Database viewer](/database-viewer): Browse schemas and table data, run read-only SQL, and inspect Redis keys for a managed database from the dashboard or the CLI.
### CI/CD and automation
* [Deploying from GitHub Actions](/github-actions): Deploy a prebuilt image from a GitHub Actions workflow with the bundled composite action, no Go toolchain needed.
* [Pipelines](/pipelines): Define CI/CD pipelines as YAML with triggers, jobs, matrix builds, approvals, and deploys, and run them on your own nodes.
* [Pipelines: OIDC federation](/pipelines-oidc): Mint short-lived OIDC tokens for pipeline jobs, and wire AWS, GCP, or Vault to trust them without a long-lived credential.
* [Git integrations](/git-integrations): Connect GitHub, GitLab, Bitbucket or Gitea for automatic git-triggered deploys, per-pull-request preview environments, and webhook history with replay.
### Security and access
* [Sign-in engine](/auth-engine): How sign-in, API tokens, two-factor, passkeys and OAuth are served, and the settings that tune them.
* [Host firewall](/host-firewall): Manage allow and deny rules for the control plane host's ufw firewall from the dashboard, CLI or API, with a lockout guard against blocking essential ports.
* [Exposure audit](/exposure-audit): Find which published container ports are reachable from the internet, why ufw does not cover them, and how to restrict one with a reversible DOCKER-USER allow-list.
* [Security center](/security-center): The security center scores your instance from its real state, lists what to fix first with a one-click fix or a link, manages every session and device, and sets sign-in approval and API token policy.
* [Emergency access](/emergency-access): Get back into a control plane you cannot sign in to by resetting the admin password, minting a short-lived root token from the host, or using a break-glass token made in advance, and what keeps each safe.
* [Master key rotation](/master-key-rotation): Rotate the envelope-encryption master key without losing access to stored secrets or causing downtime.
* [Access control](/access-control): Roles for multi-user access, per-user environment grants, the guest role, IAM policies on apps, databases and environments, and ready-made policy templates.
* [Teams and roles](/teams-and-roles): Teams own projects, environments, apps and databases. Four roles (owner, admin, developer, viewer), invitations, team scoped API tokens, protected environments and the dashboard and CLI switchers.
* [Environments](/environments): Instance-wide environments (development, test, UAT, production and custom), the dashboard environment switcher, and moving apps and databases between them with approval for protected ones.
* [Identity and access: users, roles, and IAM policies](/identity-and-access): Authentication, authorization, roles, IAM policies, and audit logging for users and API tokens.
* [Projects, organizations, and environments](/projects-and-organizations): Organize apps and databases into optional projects, organizations, and environments with shared configuration and deployment gates.
### Scale and operations
* [Managing apps at scale](/managing-apps-at-scale): Filter, tag, bulk-act on, clone and promote apps when you run dozens of them.
* [Tags](/tags): Label apps with tags such as team:payments, then filter the dashboard, CLI and API by them.
* [Command palette](/command-palette): Keyboard-first command palette and shortcuts for navigating the dashboard and running quick app actions.
* [Cost estimate](/cost-estimate): How Levelrail's "what this would cost elsewhere" app cost estimate is calculated, its assumptions, and the env vars that let you correct the reference rates for your own region.
* [Usage and cost](/usage-and-cost): Cost showback per app, project, environment and team, how your own prices and server bills are allocated by measured usage, monthly budgets with alerts, CSV export, and the limits of the numbers.
* [Integrations](/integrations): Attach curated third-party tools (PostHog, Sentry, Datadog, and more) to an app, injecting their env vars automatically at deploy time.
* [Cloud integrations](/cloud-integrations): What works today with AWS, Google Cloud, Cloudflare, DigitalOcean and Hetzner (nodes, DNS, backups, registries, events), and what is planned but not built.
* [Rolling back the control plane](/rollback): How to roll the control plane back to a previous release: what the dashboard shows and what you run on the server.
* [Resilience: what survives a control plane crash](/resilience): What actually survives when the control plane process crashes, what does not, and why, measured with a real SIGKILL against a real control plane, not assumed from the architecture.
* [Resilience, in short](/resilience-summary): A short answer to what survives a control plane crash, with a link to the full measured page.
* [Observability: metrics, logs, and alerts](/observability): Node-local metrics, logs, and alerts with federated queries, no central ingestion.
* [Public status page](/status-page): An opt-in public status page with operator-chosen components, 90 day uptime bars, incidents, maintenance announcements and an RSS feed.
* [What's new panel](/whats-new-panel): The dashboard's "What's new" panel shows recent release notes after an upgrade, sourced from the control plane's own CHANGELOG.md.
### AI and agents
* [Working with AI agents](/agents): Connect an AI coding agent to your instance, scope its token, and give it the instructions it needs to deploy safely.
* [AI control](/ai-control): Decide what AI agents and agent tokens may do on your instance, from off to admin, enforced on the server for every request.
### Templates and catalog
* [Service template catalog](/templates-and-registry): Deploy ready-made services from the built-in template catalog, save your own apps as templates, and see how templates, Compose and static sites relate.
* [Template catalog](/template-catalog): Every service in Levelrail's one-click template catalog, grouped by category, generated straight from the Go source
* [Starter kit templates](/templates): Seven starter-kit templates that show multi-service Compose wiring patterns: web plus database, worker plus queue, reverse proxy plus backends.
### Migrating from other platforms
* [Migrating from Coolify, Dokploy, or CapRover](/migrating-from-coolify-dokploy-and-caprover): Import apps, databases, env vars, domains and volumes from Coolify, Dokploy or CapRover with a dry run first, then apply
* [Migrating a Next.js app from Vercel](/migrating-from-vercel): A practical runbook for moving a Vercel-hosted Next.js app to Levelrail, with an inventory, a concept mapping table, a DNS cutover and rollback plan, the gaps that have no equivalent, and a shadow-run checklist.
* [Migrating a service from Cloud Run](/migrating-from-cloud-run): A practical runbook for moving a Google Cloud Run service to Levelrail, with a concept mapping table, deploying the same container image from a registry, secrets, Cloud SQL, domains, scheduled jobs, and an honest list of what you give up.
### Beta and experimental
* [AI assistant integration: levelrail-mcp](/ai-assistant): Run levelrail-mcp over stdio for a locally spawning client or over the network for a remotely hosted MCP-compatible AI assistant, and how to scope the API token it uses.
* [MCP tools, token budget and write mode](/mcp): The levelrail-mcp tool listing, its token budget, the read-only diagnostic tools, the confirm-gated write tools behind --allow-write, and the prompt injection threat model.
* [MCP recipes](/mcp-recipes): Five real conversations with an AI assistant connected to the MCP server, showing what you ask, which prompts and tools run, and what comes back.
* [In-app AI assistant chat](/ai-assistant-chat): The in-app AI assistant chat, how to enable it behind the ai-chat experimental flag, and the read-and-suggest confirmation gate that bounds what it can do.
* [AI models](/ai-models): Run Ollama, vLLM or llama.cpp models on NVIDIA GPU nodes as a first-class resource, with an OpenAI-compatible endpoint protected by an API key.
* [Load balancing across replicas](/load-balancing): Balance traffic across an app's replicas with health checks, retries, sticky sessions, weights, graceful cutovers, live upstream status, and Terraform, CDK, CloudFormation or Caddy export.
* [Platform as code](/platform-as-code): Describe projects, environments, apps, domains, databases and more as YAML files, review the plan, and apply it from the CLI, the dashboard, MCP or CI.
* [Resource advisor (beta)](/resource-advisor): The beta resource advisor that suggests memory and CPU limits, flags idle apps, memory growth and disk pressure, and sizes Postgres, Redis and MySQL settings from local telemetry, with every rule and threshold explained.
* [Feature flags](/feature-flags): Toggle behavior in a running app without a redeploy, with per-app flags, percentage rollouts and a simple evaluate endpoint.
* [Log archive](/log-archive): Ship container logs to storage as compressed NDJSON on a schedule.
## Reference
### Reference
* [app.yaml reference](/app-spec-reference): Complete reference for app.yaml - the declarative app spec for Levelrail
* [Feature catalog](/feature-catalog): Inventory of Levelrail's dashboard pages by area, with pointers to the generated API and CLI references.
* [Levelrail CLI Reference](/cli-reference): Complete reference of all Levelrail CLI commands, organized by group with examples and typical use cases.
* [Terminal UI](/cli-tui): Run Levelrail from the terminal with a live, keyboard-driven UI for apps, databases, deploys, nodes, domains, alerts and backups.
* [Environment variables](/environment-variables): Reference for operator-facing APP\_ environment variables not covered on feature pages, with defaults, units, and which component reads each one.
* [Environment variables and secrets](/environment-and-secrets): Masking, reveal policies, and protection for app environment variables and secrets.
* [REST API Reference](/api-reference): Complete HTTP API reference for the control plane, organized by resource group.
* [API explorer](/api-explorer): The in-dashboard API explorer under Settings, where signed-in operators browse every control plane route and try requests against their own instance.
* [Mobile API](/mobile-api): Pair a phone with the control plane: one-time pairing codes, scoped device tokens, revocation, and the single round trip overview endpoint.
* [MCP tool surface](/mcp-tool-surface): Estimated model context cost of the MCP tool list, by toolset.
* [Agent tooling audit](/agent-tooling-audit): Size, overlap and annotation audit of the levelrail-mcp tool surface, with the token budget test that keeps it from regressing.
* [Shared Go kit](/kit): Standard-library-only Go module with networking, scheduling, parsing and system helper packages.
## Concepts
### Explanation
* [Product tour](/product-tour): A look at the Levelrail dashboard: apps, deploys, logs, databases, domains and nodes, each with a short explanation.
* [Architecture](/architecture): Core architecture of Levelrail's reconciler, agent, builds, ingress, state, observability, and security layers.
* [Security overview](/security): How Levelrail handles secrets, sessions, TLS, and access control, and where to find the detailed page for each.
* [Threat model](/threat-model): Trust boundaries, assets, attackers, the mitigations that exist today with file references, known gaps, and how to report a vulnerability.
* [Docker access and the API guard](/docker-access): Every Docker Engine API call Levelrail makes, why it needs it, and the guard that refuses everything else, including privileged containers and host mounts.
* [Security alert verdicts](/security-alert-verdicts): Verdict, evidence, and dismissal command for each code scanning alert open when this page was written.
* [vs Coolify, Dokploy, CapRover, Dokku, and Kamal](/comparison): An honest comparison of Levelrail with Coolify, Dokploy, CapRover, Dokku, and Kamal: architecture, what each is good at, and what Levelrail ships without a paywall.
* [Who Levelrail is for](/use-cases): Three real audiences for Levelrail (solo developers, small agencies, teams migrating off Coolify, Dokploy, Heroku, or Vercel), what each is likely trying to do, which shipped features actually fit, and where the project is not there yet.
* [White-labeling](/white-labeling): How the brand layer works, which names and prefixes it drives (dashboard, agent unit, firewall rule tag, mesh interface, update repository), the brand.yaml fields and APP\_BRAND\_\* overrides, and what a fork must still change by hand.
* [glinr-bot](/glinr-bot): How the repo's GitHub App bot evaluates pull requests against a policy file, comments its reasoning and, in enforce mode, approves and arms auto-merge.
* [Run behind an existing proxy](/behind-an-existing-proxy): Run Levelrail behind the proxy that already owns ports 80 and 443, with managed routes for Traefik (including Coolify) and Nginx.
* [Security advisories](/security-advisories): Every security issue Levelrail has found and fixed, with a stable ID, severity, affected versions, the fix, and how we disclose.
### Compare and choose
* [Self-hosted PaaS guide: what to look for and how Levelrail fits](/self-hosted-paas): A practical guide to choosing a self-hosted PaaS: how it manages servers, deploys, rolls back, observes, secures and costs.
* [Coolify alternative: a self-hosted deploy platform that stays light](/coolify-alternative): Levelrail is an open source Coolify alternative: Docker Engine API instead of SSH, built-in metrics and logs, one-click rollback, read-only importer.
* [Dokploy alternative: self-hosted deploys without Docker Swarm](/dokploy-alternative): Levelrail is an open source Dokploy alternative that skips Docker Swarm: an agent on each node and a reconciler over the Docker Engine API.
* [CapRover alternative: self-hosted deploys without Docker Swarm](/caprover-alternative): Levelrail is an open source CapRover alternative: plain Docker instead of Swarm, an agent per node, built-in metrics and logs, one-click rollback.
* [Dokku alternative: git push deploys with a dashboard and multi-server](/dokku-alternative): Levelrail is an open source Dokku alternative that adds a dashboard, deploy history with rollback, and built-in metrics and logs.
* [Kamal alternative with a dashboard, history and built-in observability](/kamal-alternative): Levelrail is an open source Kamal alternative: the same zero-downtime idea plus a control plane, dashboard, deploy history and built-in metrics.
* [Self-hosted Heroku alternative: git push to your own servers](/heroku-alternative): Levelrail is an open source Heroku alternative for your own Linux servers: git push deploys, managed Postgres and Redis, TLS, logs and rollback.
* [Self-hosted Railway alternative with built-in observability](/railway-alternative): Levelrail is an open source Railway alternative you run on your own servers: git push deploys, managed databases, domains with TLS.
* [Self-hosted Vercel alternative for Next.js and Docker apps](/vercel-alternative): Levelrail is an open source, self-hosted Vercel alternative: git push deploys, automatic TLS and preview environments per pull request.
* [Self-hosted alternative to Cloud Run for steady workloads](/cloud-run-alternative): When an always-on container is cheaper on your own server than on Cloud Run, when it is not, and how Levelrail fits for git or image deploys.
* [Self-host Next.js on your own server with git push, TLS and rollback](/self-host-nextjs): How to self-host a Next.js app with Levelrail: Dockerfile build, automatic HTTPS, preview environments per pull request and env vars.
### Design proposals
* [Git provider integrations: shared abstraction, GHE, Bitbucket, connect-flow UX](/design/git-provider-integrations): Design for unified post-auth git provider abstraction supporting GitHub, GitLab, Bitbucket, and GitHub Enterprise Server
* [IAM: what exists and what the authoring tools add](/iam-design): How IAM policies are stored, evaluated and attached, and what the policy builder, simulator and analyzer add without changing any decision.
## Appendix
### Status and roadmap
* [Roadmap](/roadmap): What Levelrail has shipped by area, what is still open, and what is out of scope. There is no stable release yet.
* [Feature status](/feature-status): Per-feature maturity labels (stable, beta, hide-behind-flag) with the test, e2e, docs and real-infrastructure evidence behind each one.
* [Experimental features](/experimental-features): Features hidden at launch, the APP\_EXPERIMENTAL switch that turns them on, and what each surface does while a feature is off.
* [Performance](/performance): Measured idle footprint of the control plane at 0, 100, and 500 apps, and how to reproduce it.
* [Changelog](/changelog): Every Levelrail release with highlights, fixes and upgrade notes.
* [Upgrade history](/upgrade-history): Every version the control plane has run, recorded at boot, and how upgrades are acknowledged.
### Engineering notes
* [CI](/ci): How the GitHub Actions checks on a pull request are chosen, what they cost, and which ones branch protection relies on.
* [VPS smoke test](/vps-smoke): scripts/vps-smoke.sh checks a real install end to end on a VPS you already have, without creating or deleting cloud resources.
* [Internationalization (i18n)](/i18n): How the dashboard's i18n works for contributors: where strings live, how to add one to a component, and how TypeScript catches a mistyped key.
* [GLINR design tokens](/design-tokens): The single source of truth for color, space, radius, elevation, type and motion in the Levelrail dashboard, and how to add or change a token.
* [Brand kit](/brand): Logos, colors and ready-made social images for Levelrail, free to use when you write about it, share it or link to it.
### About
* [About Levelrail: open source, built by GLINCKER](/about): Levelrail is an Apache 2.0 licensed, self-hosted deployment platform built by GLINCKER. Read what it is, how it is licensed, and how pre-release it still is.
* [Pricing: Levelrail is free and open source (Apache 2.0)](/pricing): Levelrail costs nothing: Apache 2.0, one edition, no license key and no feature gated behind a plan. You only pay for your servers.
* [Levelrail demo: install, deploy and roll back in minutes](/demo): See Levelrail in action: install script, git push deploys, live logs, metrics with deploy markers, multi-node and one-click rollback.
* [Case studies: measured results, not testimonials](/case-studies): Real Levelrail measurements: a 500-app idle benchmark and multi-node verification against real Docker daemons, plus what is not proven yet.
* [Contribute to Levelrail: open source deployment platform](/get-involved): Help build Levelrail: try it and report what breaks, pick a good first issue, improve docs, add templates or test multi-node. Apache 2.0, active development.
* [Developers: API, CLI and MCP server for your Levelrail instance](/developers): Build on Levelrail: the REST API at /api/v1 on your own instance, the levelrail-cli command line, the self-hosted MCP server, and scoped API tokens.
* [Contact Levelrail: support, community and security reports](/contact): How to reach the Levelrail team: support email, GitHub issues and discussions, the Discord community, and the private security reporting process.
* [Privacy and data handling: what Levelrail stores and sends](/privacy): Where your data lives with Levelrail: secrets, metrics and logs stay on your own servers and the product sends no usage telemetry.
### Legal
* [Terms of use](/terms): Terms for using the Levelrail website and software, including the Apache 2.0 license, no warranty, trademarks and third-party links.
* [Privacy policy](/privacy-policy): What the Levelrail website collects, which browser storage it uses, and where to find how the software itself handles data.
* [Cookie notice](/cookies): The Levelrail website sets no cookies of its own and stores only your theme preference in your browser's local storage.
* [License and notices](/license-and-notices): Levelrail is licensed under Apache 2.0 with one edition and no paid tier. What that means in plain language and where to read the full text.
# Announcements
Source: https://docs.levelrail.com/announcements/index
News about Levelrail: new guides, the self-host template gallery, and platform updates.
Product news and site updates. Release notes live in the [changelog](/changelog).
* [Template gallery and guides](/announcements/self-host-template-gallery-and-guides)
# Self-host template gallery and guides are live
Source: https://docs.levelrail.com/announcements/self-host-template-gallery-and-guides
Browse 330 one-click self-host templates and read new guides on backups, zero-downtime deploys and moving from Coolify.
The docs now include a [self-host template gallery](/self-host) with a page for each of the 330 catalog templates, listing the services, ports, volumes and environment variables each one deploys.
Three new guides cover [Coolify versus Levelrail](/guides/coolify-vs-levelrail), [self-hosting Postgres with backups](/guides/self-host-postgres-with-backups) and [zero-downtime deploys without Kubernetes](/guides/zero-downtime-deploys-without-kubernetes).
You can follow along by RSS, Atom or JSON Feed from the [announcements page](/announcements).
# API explorer
Source: https://docs.levelrail.com/api-explorer
The in-dashboard API explorer under Settings, where signed-in operators browse every control plane route and try requests against their own instance.
The API explorer lists every HTTP route the control plane registers and lets you fire a real request at it from the browser. Use it to find out what the API offers, or to prototype a script or integration before writing any code.
Open **Settings > API explorer** (`/settings/api-explorer`). You need to be signed in.
## What it shows
* Every registered route, grouped by resource as in the [API reference](/api-reference).
* Each route's method, path, required ability and, where one exists, a short description.
* A filter box that matches path, method, ability and description.
* A **Try it** form per route. It sends the request to this instance with your current browser session. Path parameters such as `{name}` become text fields, and the raw JSON response is shown exactly as the API returned it. The request is real: a `POST` or `DELETE` changes your instance.
* A worked request and response for a small set of common routes (apps, brand, system status).
Most routes have no worked example and about three quarters have no description. The page header shows the current counts.
## Where the data comes from
The explorer reads `GET /api/v1/openapi.json`, which needs the `read` ability (a session or a token). Despite the name it is not a full OpenAPI document: it is a subset with method, path, ability, group and description for each route, generated from the route registrations at build time. The static [API reference](/api-reference) and the published [OpenAPI 3.1 file](/public/openapi.json) (methods, paths, path parameters and required ability, with generic bodies) are generated from the same source, so all three list the same routes.
For calling the API from outside the dashboard, create a token as described in [Identity and access](/identity-and-access). AI agents reach the same API through the [MCP tool surface](/mcp-tool-surface).
# REST API Reference
Source: https://docs.levelrail.com/api-reference
Complete HTTP API reference for the control plane, organized by resource group.
Route inventory for Levelrail's control plane HTTP API, organized by resource group matching `docs/feature-catalog.md`.
The tables below are generated from the route registrations in `internal/api/routes*.go`. After adding or changing a route, run `go run ./scripts/gen-api-reference` (fast and idempotent) and commit the result; `TestAPIReferenceCoversEveryRoute` fails if a registered route is missing. New routes land in the group with the closest path prefix, or in **Other**.
## System
System endpoints for:
* Status checks and diagnostics
* Admin operations
* Configuration retrieval and management
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /healthz | Public | handleHealthz |
| GET | /readyz | Public | handleReadyz |
| GET | /api/v1/brand | Public | handleBrand |
| GET | /api/v1/dev-mode | Public | handleDevMode |
| GET | /api/v1/system/status | AbilityRead | handleSystemStatus |
| GET | /api/v1/system/doctor | AbilityRead | handleSystemDoctor |
| GET | /api/v1/system/containers | AbilityRead | handleListContainers |
| POST | /api/v1/system/prune | AbilityRoot | handleSystemPrune |
| POST | /api/v1/system/backups | AbilityRoot | handleCreateControlPlaneBackup |
| GET | /api/v1/system/backups | AbilityRoot | handleListControlPlaneBackups |
| GET | /api/v1/system/backups//download | AbilityRoot | handleDownloadControlPlaneBackup |
| POST | /api/v1/system/backups//verify | AbilityRoot | handleVerifyControlPlaneBackup |
| DELETE | /api/v1/system/backups/ | AbilityRoot | handleDeleteControlPlaneBackup |
| GET | /api/v1/system/volumes/orphaned | AbilityRead | handleListOrphanedVolumes |
| POST | /api/v1/system/volumes/orphaned/cleanup | AbilityRoot | handleCleanupOrphanedVolumes |
| POST | /api/v1/system/master-key/rotate | AbilityRoot | handleRotateMasterKey |
| GET | /api/v1/onboarding | AbilityRead | handleGetOnboardingState |
| POST | /api/v1/onboarding/complete | AbilityWrite | handleCompleteOnboarding |
| PUT | /api/v1/onboarding/progress | AbilityRoot | handleUpdateOnboardingProgress |
| GET | /api/v1/updates | AbilityRead | handleGetUpdates |
| GET | /api/v1/system/secrets/binding | AbilityRead | handleGetSecretBinding |
| POST | /api/v1/system/secrets/rebind | AbilityRoot | handleRebindSecrets |
| GET | /api/v1/system/control-plane-dr | AbilityRead | handleGetControlPlaneDR |
| PUT | /api/v1/system/control-plane-dr/settings | AbilityWriteSensitive | handleUpdateControlPlaneDR |
| GET | /api/v1/system/control-plane-dr/backups | AbilityRead | handleListControlPlaneDRBackups |
| POST | /api/v1/system/control-plane-dr/run | AbilityWriteSensitive | handleRunControlPlaneDRBackup |
| POST | /api/v1/system/control-plane-dr/drill | AbilityWriteSensitive | handleRunControlPlaneDRDrill |
| POST | /api/v1/system/control-plane-dr/escrow | AbilityRoot | handleControlPlaneDREscrow |
| POST | /api/v1/system/control-plane-dr/escrow/ack | AbilityWriteSensitive | handleAckControlPlaneDREscrow |
| GET | /api/v1/updates/preflight | AbilityRead | handleUpdatePreflight |
| POST | /api/v1/system/containers//stop | AbilityRoot | handleStopOrphanedContainer |
| POST | /api/v1/system/containers//remove | AbilityRoot | handleRemoveOrphanedContainer |
| POST | /api/v1/system/containers//claim | AbilityWrite | handleClaimOrphanedContainer |
| GET | /api/v1/updates/settings | AbilityRoot | handleGetUpdateSettings |
| PUT | /api/v1/updates/settings | AbilityRoot | handleUpdateSettings |
| GET | /api/v1/system/orphans | AbilityRead | handleListOrphans |
| POST | /api/v1/system/orphans/reap | AbilityRoot | handleReapOrphans |
| GET | /api/v1/updates/releases | AbilityRead | handleReleaseHistory |
| GET | /api/v1/updates/rollback-plan | AbilityRoot | handleRollbackPlan |
| GET | /api/v1/updates/history | AbilityRead | handleUpgradeHistory |
| POST | /api/v1/updates/history//ack | AbilityWrite | handleAckUpgrade |
| GET | /api/v1/system/reverse-proxy | AbilityRead | handleReverseProxyGuide |
| GET | /api/v1/system/docker-guard | AbilityRead | handleGetDockerGuard |
| PUT | /api/v1/system/docker-guard | AbilityRoot | handleUpdateDockerGuard |
| GET | /api/v1/system/geoip | AbilityRead | handleGeoIPLookup |
| GET | /api/v1/system/proxy-integration | AbilityRead | handleGetProxyIntegration |
| POST | /api/v1/system/proxy-integration/setup | AbilityRoot | handleSetupProxyIntegration |
| POST | /api/v1/system/proxy-integration/apply | AbilityRoot | handleApplyProxyIntegration |
| POST | /api/v1/system/proxy-integration/verify | AbilityRoot | handleVerifyProxyIntegration |
| GET | /api/v1/system/readiness | AbilityRead | handleServerReadiness |
| GET | /api/v1/updates/self-upgrade/plan | AbilityRead | handleSelfUpgradePlan |
| GET | /api/v1/updates/self-upgrade/attempts | AbilityRead | handleSelfUpgradeAttempts |
| POST | /api/v1/updates/self-upgrade | AbilityRoot | handleSelfUpgradeStart |
| GET | /api/v1/updates/schedule | AbilityRead | handleGetUpgradeSchedule |
| POST | /api/v1/updates/schedule | AbilityRoot | handleCreateUpgradeSchedule |
| DELETE | /api/v1/updates/schedule | AbilityRoot | handleCancelUpgradeSchedule |
| GET | /api/v1/updates/auto | AbilityRoot | handleGetUpgradePolicy |
| PUT | /api/v1/updates/auto | AbilityRoot | handleUpdateUpgradePolicy |
| POST | /api/v1/updates/backup-now | AbilityRoot | handleUpgradeBackupNow |
| GET | /api/v1/system/server-address | AbilityRead | handleGetServerAddress |
| PUT | /api/v1/system/server-address | AbilityRoot | handleSetServerAddress |
| GET | /api/v1/system/restart | AbilityRead | handleGetSystemRestart |
| POST | /api/v1/system/restart | AbilityRoot | handleSystemRestart |
| GET | /api/v1/system/cli-console/commands | AbilityRoot | handleCLIConsoleCommands |
| GET | /api/v1/system/cli-console/session | AbilityRoot | handleCLIConsoleSession |
| GET | /api/v1/system/networks | AbilityRead | handleGetSystemNetworks |
| POST | /api/v1/system/networks/prune | AbilityRoot | handlePruneSystemNetworks |
## Auth / 2FA / Users / Roles / IAM / Device Auth / OAuth
Endpoints for:
* Authentication and session management
* Two-factor authentication setup and verification
* User and role management
* IAM policies and attachment
* Device login flows
* OAuth provider configuration
| Method | Path | Ability | Handler |
| - | - | - | - |
| POST | /api/v1/auth/login | Public | handleLogin |
| POST | /api/v1/auth/register | Public (setup token) | handleRegister |
| GET | /api/v1/auth/setup-status | Public | handleSetupStatus |
| POST | /api/v1/auth/logout | Session | handleLogout |
| PUT | /api/v1/auth/password | Session | handleChangePassword |
| GET | /api/v1/auth/session | Session | handleGetSession |
| POST | /api/v1/auth/sessions/revoke-others | Session | handleRevokeOtherSessions |
| POST | /api/v1/auth/users | AbilityRoot | handleCreateUser |
| GET | /api/v1/users | AbilityRead | handleListUsers |
| DELETE | /api/v1/users/ | AbilityRoot | handleDeleteUser |
| PUT | /api/v1/users//abilities | AbilityRoot | handleUpdateUserAbilities |
| GET | /api/v1/roles | AbilityRead | handleListRoles |
| POST | /api/v1/invites | AbilityWrite | handleCreateInvite |
| GET | /api/v1/invites | AbilityRead | handleListInvites |
| DELETE | /api/v1/invites/ | AbilityWrite | handleRevokeInvite |
| POST | /api/v1/invites/accept | Public | handleAcceptInvite |
| POST | /api/v1/iam/policies | AbilityRoot | handleCreatePolicy |
| GET | /api/v1/iam/policies | AbilityRead | handleListPolicies |
| GET | /api/v1/iam/policies/ | AbilityRead | handleGetPolicy |
| PUT | /api/v1/iam/policies/ | AbilityRoot | handleUpdatePolicy |
| DELETE | /api/v1/iam/policies/ | AbilityRoot | handleDeletePolicy |
| GET | /api/v1/iam/policies//attachments | AbilityRead | handleListPolicyAttachments |
| POST | /api/v1/iam/policies//attachments | AbilityRoot | handleAttachPolicy |
| DELETE | /api/v1/iam/policies//attachments// | AbilityRoot | handleDetachPolicy |
| POST | /api/v1/auth/device/start | Public | handleDeviceAuthStart |
| POST | /api/v1/auth/device/token | Public | handleDeviceAuthToken |
| GET | /api/v1/auth/device/requests | Session | handleListDeviceAuthRequests |
| POST | /api/v1/auth/device//approve | Session | handleApproveDeviceAuthRequest |
| POST | /api/v1/auth/device//deny | Session | handleDenyDeviceAuthRequest |
| GET | /api/v1/auth/oauth/providers | Public | handleListPublicOAuthProviders |
| GET | /api/v1/auth/oauth//start | Public | handleOAuthStart |
| GET | /api/v1/auth/oauth//callback | Public | handleOAuthCallback |
| GET | /api/v1/auth/oauth//link/start | Session | handleOAuthLinkStart |
| GET | /api/v1/settings/oauth | AbilityRead | handleListOAuthSettings |
| PUT | /api/v1/settings/oauth/ | AbilityRoot | handleUpdateOAuthProviderSettings |
| POST | /api/v1/auth/forgot-password | Public | handleForgotPassword |
| POST | /api/v1/auth/reset-password | Public | handleResetPassword |
| POST | /api/v1/auth/tokens | Session | handleCreateToken |
| GET | /api/v1/auth/tokens | Session | handleListTokens |
| DELETE | /api/v1/auth/tokens/ | Session | handleRevokeToken |
| GET | /api/v1/settings/ai-assistant | AbilityRead | handleGetAIAssistantSettings |
| PUT | /api/v1/settings/ai-assistant | AbilityRoot | handleUpdateAIAssistantSettings |
| GET | /api/v1/settings/deploy-freeze | AbilityRead | handleGetGlobalDeployFreeze |
| PUT | /api/v1/settings/deploy-freeze | AbilityRoot | handlePutGlobalDeployFreeze |
| GET | /api/v1/settings/push-subscriptions/vapid-public-key | Session | handleGetPushVAPIDPublicKey |
| GET | /api/v1/settings/push-subscriptions | Session | handleListPushSubscriptions |
| POST | /api/v1/settings/push-subscriptions | Session | handleCreatePushSubscription |
| DELETE | /api/v1/settings/push-subscriptions/ | Session | handleDeletePushSubscription |
| GET | /api/v1/settings/observability | AbilityRead | handleGetObservabilitySettings |
| PUT | /api/v1/settings/observability | AbilityRoot | handleUpdateObservabilitySettings |
| POST | /api/v1/auth/session-links | AbilityRoot | handleMintSessionLink |
| GET | /api/v1/auth/session-links//consume | Public | handleConsumeSessionLinkGET |
| GET | /api/v1/auth/whoami | Public | handleWhoami |
| POST | /api/v1/roles | AbilityRoot | handleCreateRole |
| PUT | /api/v1/roles/ | AbilityRoot | handleUpdateRole |
| DELETE | /api/v1/roles/ | AbilityRoot | handleDeleteRole |
| PUT | /api/v1/users//role | AbilityRoot | handleSetUserRole |
| GET | /api/v1/users//environment-grants | AbilityRoot | handleGetEnvironmentGrants |
| PUT | /api/v1/users//environment-grants | AbilityRoot | handlePutEnvironmentGrants |
| GET | /api/v1/iam/policy-templates | AbilityRead | handleListPolicyTemplates |
| POST | /api/v1/iam/policy-templates//apply | AbilityRoot | handleApplyPolicyTemplate |
| GET | /api/v1/settings/ai-control | AbilityRead | handleGetAIControl |
| PUT | /api/v1/settings/ai-control | AbilityRoot | handleUpdateAIControl |
| POST | /api/v1/settings/ai-control/revoke-agent-tokens | AbilityRoot | handleRevokeAgentTokens |
| GET | /api/v1/auth/device/pending-summary | AbilityRead | handleDevicePendingSummary |
| GET | /api/v1/auth/device/activity | AbilityRead | handleDeviceActivity |
| GET | /api/v1/iam/catalog | AbilityRead | handleIAMCatalog |
| GET | /api/v1/iam/resources | AbilityRead | handleIAMResources |
| POST | /api/v1/iam/resources/match | AbilityRead | handleIAMMatch |
| GET | /api/v1/iam/principals | AbilityRead | handleIAMPrincipals |
| GET | /api/v1/iam/principals///effective | AbilityRead | handleIAMEffective |
| GET | /api/v1/iam/simulate | AbilityRead | handleIAMSimulate |
| GET | /api/v1/iam/analyze | AbilityRead | handleIAMAnalyze |
| POST | /api/v1/iam/policies/validate | AbilityRead | handleValidatePolicy |
| POST | /api/v1/iam/preview | AbilityRead | handleIAMPreview |
| GET | /api/v1/iam/policies//versions | AbilityRead | handleListPolicyVersions |
| POST | /api/v1/iam/policy-templates//render | AbilityRead | handleRenderPolicyTemplate |
| GET | /api/v1/auth/login-options | Public | handleLoginOptions |
| POST | /api/v1/auth/login-code/request | Public | handleRequestLoginCode |
| POST | /api/v1/auth/login-code/redeem | Public | handleRedeemLoginCode |
| POST | /api/v1/auth/login-approval/poll | Public | handlePollLoginApproval |
| GET | /api/v1/auth/sign-in-requests | Public | handleListSignInRequests |
| POST | /api/v1/auth/sign-in-requests/codes//reveal | Public | handleRevealLoginCode |
| POST | /api/v1/auth/login-approvals//approve | Public | handleApproveLoginApproval |
| POST | /api/v1/auth/login-approvals//deny | Public | handleDenyLoginApproval |
| GET | /api/v1/auth/trusted-devices | Public | handleListTrustedDevices |
| DELETE | /api/v1/auth/trusted-devices/ | Public | handleRevokeTrustedDevice |
| GET | /api/v1/settings/auth/code-login | AbilityRead | handleGetCodeLoginSettings |
| PUT | /api/v1/settings/auth/code-login | AbilityRoot | handlePutCodeLoginSettings |
| POST | /api/v1/auth/sign-in-alert/disown | Public | handleDisownSignIn |
| GET | /api/v1/security/posture | AbilityRead | handleSecurityPosture |
| GET | /api/v1/security/policy | AbilityRead | handleGetSecurityPolicy |
| PUT | /api/v1/security/policy | AbilityRoot | handlePutSecurityPolicy |
| GET | /api/v1/security/sessions | Public | handleListSecuritySessions |
| DELETE | /api/v1/security/sessions/ | Public | handleRevokeSecuritySession |
| POST | /api/v1/security/sessions/revoke-others | Public | handleRevokeOtherSecuritySessions |
| GET | /api/v1/security/account | Public | handleGetAccountSecurity |
| PUT | /api/v1/security/account | Public | handlePutAccountSecurity |
| GET | /api/v1/settings/database-upgrades | AbilityRead | handleGetPlatformUpgradePolicy |
| PUT | /api/v1/settings/database-upgrades | AbilityRoot | handlePutPlatformUpgradePolicy |
| GET | /api/v1/settings/domain-automation | AbilityRead | handleGetDomainAutomation |
| PUT | /api/v1/settings/domain-automation | AbilityRoot | handleUpdateDomainAutomation |
| PUT | /api/v1/settings/proxy-integration | AbilityRoot | handleUpdateProxyIntegrationSettings |
| GET | /api/v1/auth/step-up | Public | handleStepUpOptions |
| POST | /api/v1/auth/step-up | Public | handleStepUp |
| POST | /api/v1/auth/step-up/passkey/begin | Public | handleStepUpPasskeyBegin |
| POST | /api/v1/auth/step-up/passkey/finish | Public | handleStepUpPasskeyFinish |
| GET | /api/v1/settings/beta-features | AbilityRead | handleListBetaFeatures |
| PUT | /api/v1/settings/beta-features/ | AbilityRoot | handleSetBetaFeature |
| POST | /api/v1/auth/session-links/consume | Public | handleConsumeSessionLink |
| PATCH | /api/v1/auth/profile | Session | handleUpdateProfile |
| GET | /api/v1/auth/my-access | Public | handleMyAccess |
| GET | /api/v1/auth/saml/providers | Public | handleListPublicSAMLProviders |
| GET | /api/v1/auth/saml//metadata | Public | handleSAMLMetadata |
| GET | /api/v1/auth/saml//login | Public | handleSAMLLogin |
| POST | /api/v1/auth/saml//acs | Public | handleSAMLACS |
| GET | /api/v1/security/session-policies | AbilityRoot | handleListSessionPolicies |
| PUT | /api/v1/security/session-policies/ | AbilityRoot | handlePutSessionPolicy |
| DELETE | /api/v1/security/session-policies/ | AbilityRoot | handleDeleteSessionPolicy |
| GET | /api/v1/security/session-policy/effective | Public | handleEffectiveSessionPolicy |
| PUT | /api/v1/settings/database-catalog | AbilityRoot | handlePutDatabaseCatalogSettings |
| PUT | /api/v1/users//password-change-required | AbilityRoot | handleSetPasswordChangeRequired |
| GET | /api/v1/settings/ai-assistant/providers | AbilityRead | handleListAIProviders |
| PUT | /api/v1/settings/ai-assistant/providers/ | AbilityRoot | handleSetAIProvider |
| POST | /api/v1/settings/ai-assistant/providers//test | AbilityRoot | handleTestAIProvider |
| DELETE | /api/v1/settings/ai-assistant/providers/ | AbilityRoot | handleDeleteAIProvider |
| GET | /api/v1/settings/ai-assistant/providers//models | AbilityRoot | handleListAIProviderModels |
| GET | /api/v1/auth/2fa | Session | |
| POST | /api/v1/auth/2fa/setup | Session | |
| POST | /api/v1/auth/2fa/confirm | Session | |
| POST | /api/v1/auth/2fa/disable | Session | |
| POST | /api/v1/auth/2fa/recovery-codes/regenerate | Session | |
| POST | /api/v1/auth/2fa/verify | Public | |
| GET | /api/v1/auth/passkeys | Session | |
| POST | /api/v1/auth/passkeys/register/begin | Session | |
| POST | /api/v1/auth/passkeys/register/finish | Session | |
| DELETE | /api/v1/auth/passkeys/ | Session | |
| PATCH | /api/v1/auth/passkeys/ | Session | |
| POST | /api/v1/auth/passkey-login/begin | Public | |
| POST | /api/v1/auth/passkey-login/finish | Public | |
## Apps CRUD / Lifecycle / Deploy
Endpoints for:
* Application creation, retrieval, update, and deletion
* Deployment triggers and history
* Application control (restart, stop, start)
* Logs, metrics, and diagnostics
* Build operations and deployment comparisons
* Image and network information
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/apps | AbilityRead | handleListApps |
| POST | /api/v1/apps | AbilityWrite | handleCreateApp |
| GET | /api/v1/apps/ | AbilityRead | handleGetApp |
| PUT | /api/v1/apps/ | AbilityWrite | handleUpdateApp |
| DELETE | /api/v1/apps/ | AbilityWrite | handleDeleteApp |
| GET | /api/v1/apps//health | AbilityRead | handleGetAppHealth |
| PUT | /api/v1/apps//health | AbilityWrite | handleSetAppHealth |
| DELETE | /api/v1/apps//health | AbilityWrite | handleClearAppHealth |
| GET | /api/v1/apps//group | AbilityRead | handleGetAppGroup |
| GET | /api/v1/apps//hook-runs | AbilityRead | handleGetAppHookRuns |
| POST | /api/v1/apps//compose | AbilityDeploy | handleDeployCompose |
| GET | /api/v1/service-templates | AbilityRead | handleListServiceTemplates |
| GET | /api/v1/service-templates/ | AbilityRead | handleGetServiceTemplate |
| POST | /api/v1/apps//clone | AbilityWrite | handleCloneApp |
| PUT | /api/v1/apps//node | AbilityRoot | handleSetAppNode |
| POST | /api/v1/apps//deploys | AbilityDeploy | handleTriggerDeploy |
| GET | /api/v1/apps//deploys | AbilityRead | handleDeployHistory |
| POST | /api/v1/apps//restart | AbilityDeploy | handleRestartApp |
| POST | /api/v1/apps//stop | AbilityDeploy | handleStopApp |
| POST | /api/v1/apps//start | AbilityDeploy | handleStartApp |
| POST | /api/v1/apps//exec | AbilityRoot | handleExecApp |
| GET | /api/v1/apps//terminal | AbilityRoot | handleAppTerminal |
| GET | /api/v1/apps//deploy-attempts | AbilityRead | handleListDeployAttempts |
| GET | /api/v1/deploys/failed | AbilityRead | handleListFailedDeploys |
| GET | /api/v1/apps//deploys/compare | AbilityRead | handleCompareDeploys |
| GET | /api/v1/apps//promote/preview | AbilityRead | handlePromotePreview |
| POST | /api/v1/apps//promote | AbilityDeploy | handlePromoteApp |
| GET | /api/v1/apps//deploys//logs | AbilityRead | handleDeployLogStream |
| GET | /api/v1/apps//deploys//logs/download | AbilityRead | handleDownloadDeployLog |
| GET | /api/v1/apps//diagnose | AbilityRead | handleDiagnoseApp |
| GET | /api/v1/apps//resource-recommendation | AbilityRead | handleAppResourceRecommendation |
| POST | /api/v1/apps//builds | AbilityDeploy | handleTriggerBuild |
| POST | /api/v1/apps//deploy-spec | AbilityDeploy | handleDeploySpec |
| POST | /api/v1/git/branches | AbilityDeploy | handleListGitBranches |
| GET | /api/v1/apps//images | AbilityRead | handleListImages |
| GET | /api/v1/apps//network | AbilityRead | handleGetAppNetwork |
| POST | /api/v1/apps//move-with-volumes | AbilityRoot | handleMoveAppWithVolumes |
| GET | /api/v1/apps//moves | AbilityRead | handleListAppVolumeMoves |
| GET | /api/v1/apps//moves/ | AbilityRead | handleGetAppVolumeMove |
| GET | /api/v1/apps//auto-rollback | AbilityRead | handleGetAutoRollback |
| PUT | /api/v1/apps//auto-rollback | AbilityDeploy | handleSetAutoRollback |
| GET | /api/v1/apps//auto-rollback-slo-burn | AbilityRead | handleGetAutoRollbackSLOBurn |
| PUT | /api/v1/apps//auto-rollback-slo-burn | AbilityDeploy | handleSetAutoRollbackSLOBurn |
| GET | /api/v1/apps//exec-access | AbilityRead | handleGetExecAccess |
| PUT | /api/v1/apps//exec-access | AbilityRoot | handleSetExecAccess |
| GET | /api/v1/apps//deploys//steps | AbilityRead | handleDeployStepStream |
| GET | /api/v1/apps//tags | AbilityRead | handleListAppTags |
| POST | /api/v1/apps//tags | AbilityWrite | handleAttachAppTag |
| DELETE | /api/v1/apps//tags/ | AbilityWrite | handleDetachAppTag |
| GET | /api/v1/apps//integrations | AbilityRead | handleListAppIntegrations |
| POST | /api/v1/apps//integrations | AbilityWriteSensitive | handleAttachAppIntegration |
| DELETE | /api/v1/apps//integrations/ | AbilityWriteSensitive | handleDetachAppIntegration |
| GET | /api/v1/apps//egress-policy | AbilityRead | handleGetAppEgressPolicy |
| PUT | /api/v1/apps//egress-policy | AbilityWriteSensitive | handleSetAppEgressPolicy |
| DELETE | /api/v1/apps//egress-policy | AbilityWriteSensitive | handleClearAppEgressPolicy |
| GET | /api/v1/apps//pipelines | AbilityRead | handleListPipelines |
| POST | /api/v1/apps//pipelines | AbilityWrite | handleCreatePipeline |
| GET | /api/v1/apps//pipelines/ | AbilityRead | handleGetPipeline |
| PUT | /api/v1/apps//pipelines/ | AbilityWrite | handleUpdatePipeline |
| DELETE | /api/v1/apps//pipelines/ | AbilityWrite | handleDeletePipeline |
| POST | /api/v1/apps//pipelines//runs | AbilityDeploy | handleStartPipelineRun |
| GET | /api/v1/apps//pipeline-runs | AbilityRead | handleListPipelineRuns |
| GET | /api/v1/apps//pipeline-runs/ | AbilityRead | handleGetPipelineRun |
| GET | /api/v1/apps//pipeline-runs//logs | AbilityRead | handleListPipelineRunLogs |
| GET | /api/v1/apps//pipeline-runs//logs/stream | AbilityRead | handleStreamPipelineRunLogs |
| POST | /api/v1/apps//pipeline-runs//cancel | AbilityDeploy | handleCancelPipelineRun |
| POST | /api/v1/apps//pipeline-runs//rerun | AbilityDeploy | handleRerunPipelineRun |
| POST | /api/v1/apps//pipeline-runs//approvals/ | AbilityDeploy | handleDecidePipelineApproval |
| GET | /api/v1/apps//loadbalancer | AbilityRead | handleGetLoadBalancer |
| PUT | /api/v1/apps//loadbalancer | AbilityWrite | handleSetLoadBalancer |
| DELETE | /api/v1/apps//loadbalancer | AbilityWrite | handleDeleteLoadBalancer |
| POST | /api/v1/apps//loadbalancer/import | AbilityWrite | handleImportLoadBalancer |
| GET | /api/v1/apps//loadbalancer/status | AbilityRead | handleLoadBalancerStatus |
| GET | /api/v1/apps//loadbalancer/export | AbilityRead | handleExportLoadBalancer |
| POST | /api/v1/apps//pipeline-runs//hold | AbilityDeploy | handleDecidePipelineRunHold |
| GET | /api/v1/apps//pipeline-triggers | AbilityRead | handleListPipelineTriggers |
| GET | /api/v1/apps//pipeline-sync | AbilityRead | handleGetPipelineSync |
| PUT | /api/v1/apps//pipeline-sync | AbilityWrite | handleSetPipelineSync |
| POST | /api/v1/apps//pipeline-sync | AbilityWrite | handleRunPipelineSync |
| GET | /api/v1/apps//alert-history | AbilityRead | handleListAppAlertHistory |
| POST | /api/v1/apps/bulk | AbilityWrite | handleBulkApps |
| GET | /api/v1/apps//clone/preview | AbilityRead | handleClonePreview |
| GET | /api/v1/apps//deploy-freeze | AbilityRead | handleGetAppDeployFreeze |
| PUT | /api/v1/apps//deploy-freeze | AbilityDeploy | handlePutAppDeployFreeze |
| POST | /api/v1/apps//preflight | AbilityRead | handlePreflightApp |
| GET | /api/v1/apps//requests | AbilityRead | handleQueryRequests |
| GET | /api/v1/apps//loadbalancer/history | AbilityRead | handleLoadBalancerHistory |
| POST | /api/v1/apps//loadbalancer/check | AbilityWrite | handleLoadBalancerCheck |
| PUT | /api/v1/apps//loadbalancer/upstreams/ | AbilityWrite | handleSetLoadBalancerUpstream |
| GET | /api/v1/apps//timeline | AbilityRead | handleAppTimeline |
| GET | /api/v1/apps//pending-changes | AbilityRead | handlePendingChanges |
| POST | /api/v1/apps//apply-pending | AbilityDeploy | handleApplyPending |
| GET | /api/v1/apps//preview | AbilityRead | handleGetPreview |
| PUT | /api/v1/apps//preview | AbilityWrite | handlePutPreview |
| POST | /api/v1/apps//preview/capture | AbilityWrite | handleCapturePreview |
| POST | /api/v1/apps//preview/prune | AbilityWrite | handlePrunePreview |
| GET | /api/v1/apps//preview/history | AbilityRead | handleListPreviews |
| GET | /api/v1/apps//deployments//preview | AbilityRead | handleGetPreviewImage |
| GET | /api/v1/apps//changes | AbilityRead | handleAppChanges |
| GET | /api/v1/apps//slo-preview | AbilityRead | handleSLOPreview |
| GET | /api/v1/apps//supply-chain | AbilityRead | handleGetSupplyChain |
| PUT | /api/v1/apps//supply-chain | AbilityWrite | handlePutSupplyChain |
| POST | /api/v1/apps//supply-chain/override | AbilityWrite | handleSupplyChainOverride |
| GET | /api/v1/apps//deployments//sbom | AbilityRead | handleGetSBOM |
| GET | /api/v1/apps//deployments//vulnerabilities | AbilityRead | handleGetVulnerabilities |
| POST | /api/v1/apps//deployments//scan | AbilityWrite | handleScanDeployment |
| POST | /api/v1/apps//deploys//cancel | AbilityDeploy | handleCancelDeploy |
| POST | /api/v1/apps//deploys//rollback | AbilityDeploy | handleRollbackToDeploy |
| GET | /api/v1/apps//cancel-superseded | AbilityRead | handleGetCancelSuperseded |
| PUT | /api/v1/apps//cancel-superseded | AbilityDeploy | handleSetCancelSuperseded |
| GET | /api/v1/apps//preview-policy | AbilityRead | handleGetPreviewPolicy |
| PUT | /api/v1/apps//preview-policy | AbilityWriteSensitive | handleSetPreviewPolicy |
| GET | /api/v1/apps//deploys/ | AbilityRead | handleGetDeploy |
| GET | /api/v1/apps//schedule | AbilityRead | handleGetAppSchedule |
| PUT | /api/v1/apps//schedule | AbilityWriteSensitive | handleSetAppSchedule |
| GET | /api/v1/apps//schedule/history | AbilityRead | handleListAppScheduleHistory |
| POST | /api/v1/service-templates//deploy | AbilityDeploy | handleDeployServiceTemplateNow |
| GET | /api/v1/apps//connections | AbilityRead | handleListAppConnections |
| POST | /api/v1/apps//connections | AbilityWrite | handleCreateAppConnection |
| DELETE | /api/v1/apps//connections/ | AbilityWrite | handleDeleteAppConnection |
| GET | /api/v1/apps//connectable-databases | AbilityRead | handleListConnectableDatabases |
| POST | /api/v1/apps//validate-spec | AbilityRead | handleValidateSpec |
| GET | /api/v1/apps//health-score | AbilityRead | handleGetAppHealthScore |
| POST | /api/v1/apps//save-as-template | AbilityWrite | handleSaveAppAsTemplate |
| GET | /api/v1/apps//cost-estimate | AbilityRead | handleAppCostEstimate |
| GET | /api/v1/apps//badge | AbilityRead | handleGetBadgeSettings |
| PUT | /api/v1/apps//badge | AbilityRoot | handleSetBadgeSettings |
| GET | /api/v1/apps//badge.svg | Public | handlePublicAppBadge |
| GET | /api/v1/apps//deploys//probes | AbilityRead | handleListProbeAttempts |
| POST | /api/v1/apps//health/discover | AbilityRead | handleDiscoverAppHealth |
| GET | /api/v1/apps//streams | AbilityRead | handleListAppStreams |
| POST | /api/v1/apps//streams | AbilityWriteSensitive | handleCreateAppStream |
| DELETE | /api/v1/apps//streams/ | AbilityWriteSensitive | handleDeleteAppStream |
| GET | /api/v1/apps/git-sources | AbilityRead | handleListGitSources |
| GET | /api/v1/apps//auto-update | AbilityRead | handleGetImageAutoUpdate |
| PUT | /api/v1/apps//auto-update | AbilityDeploy | handleSetImageAutoUpdate |
| POST | /api/v1/apps//auto-update/check | AbilityDeploy | handleCheckImageAutoUpdate |
| POST | /api/v1/apps//auto-update/webhook | AbilityWriteSensitive | handleRotateImageUpdateWebhook |
| GET | /api/v1/apps//canary | AbilityRead | handleGetCanary |
| POST | /api/v1/apps//canary | AbilityDeploy | handleStartCanary |
| PUT | /api/v1/apps//canary | AbilityDeploy | handleSetCanaryWeight |
| DELETE | /api/v1/apps//canary | AbilityDeploy | handleAbortCanary |
| POST | /api/v1/apps//canary/promote | AbilityDeploy | handlePromoteCanary |
| GET | /api/v1/apps//sleep | AbilityRead | handleGetAppSleep |
| PUT | /api/v1/apps//sleep | AbilityDeploy | handleSetAppSleep |
| POST | /api/v1/apps//sleep/wake | AbilityDeploy | handleWakeApp |
| GET | /api/v1/apps//environment-domains | AbilityRead | handleGetAppEnvironmentDomains |
| GET | /api/v1/apps//listening-ports | AbilityRead | handleAppListeningPorts |
| GET | /api/v1/apps//compose | AbilityRead | handleGetComposeStack |
| GET | /api/v1/apps//compose/revisions | AbilityRead | handleListComposeRevisions |
| GET | /api/v1/apps//compose/revisions/ | AbilityRead | handleGetComposeRevision |
| POST | /api/v1/apps//compose/rollback | AbilityDeploy | handleRollbackCompose |
| POST | /api/v1/apps//compose/redeploy | AbilityDeploy | handleRedeployCompose |
| POST | /api/v1/service-templates//install | AbilityDeploy | handleInstallServiceTemplate |
| DELETE | /api/v1/apps//compose | AbilityWrite | handleComposeDown |
| POST | /api/v1/apps//move | AbilityRoot | handleStartMove |
| GET | /api/v1/apps//placement | AbilityRead | handleGetPlacement |
| PUT | /api/v1/apps//placement | AbilityRoot | handleSetPlacement |
| GET | /api/v1/apps//investigate | AbilityRead | handleInvestigate |
| GET | /api/v1/apps//failure-context | AbilityRead | handleFailureContext |
| GET | /api/v1/apps//deploys//gate | AbilityRead | handleGetDeployGate |
| GET | /api/v1/apps//deploy-safety | AbilityRead | handleGetDeploySafety |
| PUT | /api/v1/apps//deploy-safety | AbilityDeploy | handlePutDeploySafety |
| GET | /api/v1/apps//rollback-targets | AbilityRead | handleListRollbackTargets |
| GET | /api/v1/apps//job-runs | AbilityRead | handleListScheduledTaskRuns |
| POST | /api/v1/apps//env/reveal | AbilityReadSensitive | handleRevealEnv |
| POST | /api/v1/apps//env/export | AbilityReadSensitive | handleExportEnvRevealed |
| GET | /api/v1/apps//env/export | AbilityRead | handleExportEnvRedacted |
| GET | /api/v1/apps//env/settings | AbilityRead | handleGetEnvSettings |
| PUT | /api/v1/apps//env/settings | AbilityWriteSensitive | handlePutEnvSettings |
| POST | /api/v1/apps//env//convert-to-secret | AbilityWriteSensitive | handleConvertEnvToSecret |
| GET | /api/v1/apps//secret-rotation | AbilityRead | handleListSecretRotation |
| GET | /api/v1/apps//secret-scan | AbilityRead | handleGetSecretScan |
| PUT | /api/v1/apps//secret-scan | AbilityWrite | handlePutSecretScan |
| POST | /api/v1/apps//secret-scan/run | AbilityWrite | handleRunSecretScan |
| POST | /api/v1/apps//secret-scan/allow | AbilityWrite | handleAllowSecretScan |
| DELETE | /api/v1/apps//secret-scan/allow/ | AbilityWrite | handleUnallowSecretScan |
| GET | /api/v1/apps//deployments//secret-scan | AbilityRead | handleGetDeploymentSecretScan |
| POST | /api/v1/apps//keep | AbilityWrite | handleKeepTrialApp |
| POST | /api/v1/apps//sleep/now | AbilityDeploy | handleSleepAppNow |
| GET | /api/v1/apps//tracing | AbilityRead | handleGetTracing |
| POST | /api/v1/apps//tracing | AbilityWriteSensitive | handleEnableTracing |
| DELETE | /api/v1/apps//tracing | AbilityWriteSensitive | handleDisableTracing |
| GET | /api/v1/apps//traces | AbilityRead | handleListTraces |
| GET | /api/v1/apps//traces/ | AbilityRead | handleGetTrace |
| GET | /api/v1/apps//log-pipeline | AbilityRead | handleGetLogPipeline |
| PUT | /api/v1/apps//log-pipeline | AbilityWriteSensitive | handleSetLogPipeline |
| DELETE | /api/v1/apps//log-pipeline | AbilityWriteSensitive | handleDeleteLogPipeline |
| POST | /api/v1/apps//log-pipeline/preview | AbilityRead | handlePreviewLogPipeline |
| GET | /api/v1/apps//access | AbilityRead | handleAppAccess |
| GET | /api/v1/apps//error-tracking | AbilityRead | handleGetErrorTracking |
| POST | /api/v1/apps//error-tracking | AbilityWriteSensitive | handleEnableErrorTracking |
| DELETE | /api/v1/apps//error-tracking | AbilityWriteSensitive | handleDisableErrorTracking |
| GET | /api/v1/apps//errors | AbilityRead | handleListErrors |
| GET | /api/v1/apps//errors/ | AbilityRead | handleGetError |
| PATCH | /api/v1/apps//errors/ | AbilityWriteSensitive | handlePatchError |
| GET | /api/v1/apps//log-budget | AbilityRead | handleGetLogBudget |
| PUT | /api/v1/apps//log-budget | AbilityWriteSensitive | handleSetLogBudget |
| DELETE | /api/v1/apps//log-budget | AbilityWriteSensitive | handleDeleteLogBudget |
| GET | /api/v1/apps//exposure | AbilityRead | handleGetAppExposure |
| POST | /api/v1/apps//exposure/preview | AbilityRead | handlePreviewAppExposure |
| PUT | /api/v1/apps//exposure | AbilityWriteSensitive | handleSetAppExposure |
| POST | /api/v1/apps//exposure/check | AbilityRead | handleCheckAppExposure |
| GET | /api/v1/apps//urls | AbilityRead | handleGetAppURLs |
| GET | /api/v1/apps//sleep/history | AbilityRead | handleAppSleepHistory |
| GET | /api/v1/apps//deploys/release-notes | AbilityRead | handleReleaseNotes |
| GET | /api/v1/apps//build-stats | AbilityRead | handleBuildStats |
| GET | /api/v1/apps//github-actions | AbilityRead | handleGetAppGitHubActions |
| PUT | /api/v1/apps//github-actions | AbilityWriteSensitive | handleSetAppGitHubActions |
| GET | /api/v1/apps/sleep/overview | AbilityRead | handleAppsSleepOverview |
| GET | /api/v1/apps//pipeline-runs//artifacts | AbilityRead | handleListPipelineRunArtifacts |
| GET | /api/v1/apps//pipeline-runs//artifacts//download | AbilityRead | handleDownloadPipelineArtifact |
| GET | /api/v1/apps//pipelines//webhook | AbilityRead | handleGetPipelineWebhook |
| PUT | /api/v1/apps//pipelines//webhook | AbilityWriteSensitive | handleSetPipelineWebhook |
| DELETE | /api/v1/apps//pipelines//webhook | AbilityWriteSensitive | handleDeletePipelineWebhook |
| POST | /api/v1/apps//pipelines//webhook/rotate | AbilityWriteSensitive | handleRotatePipelineWebhook |
| GET | /api/v1/apps//pipelines//webhook/deliveries | AbilityRead | handleListPipelineWebhookDeliveries |
| POST | /api/v1/apps//pipelines//webhook/deliveries//replay | AbilityDeploy | handleReplayPipelineWebhookDelivery |
| GET | /api/v1/apps//maintenance | AbilityRead | handleGetAppMaintenance |
| PUT | /api/v1/apps//maintenance | AbilityDeploy | handleSetAppMaintenance |
| DELETE | /api/v1/apps//maintenance | AbilityDeploy | handleClearAppMaintenance |
`PUT /api/v1/apps/{name}/health` replaces only the app's health config (the body is a `ServiceHealth`: `readiness`, `liveness`, `ready_timeout`), leaving every other field alone; an empty body clears it. Probe fields and validation match [the app spec](/app-spec-reference#probe), with durations in nanoseconds.
## Secrets / Git Source / Webhooks / Preview Environments
Endpoints for:
* Encrypted secrets management and locking
* Git repository connections and configuration
* Webhook delivery history and replay
* [Preview environment](/previews) configuration and lifecycle
| Method | Path | Ability | Handler |
| - | - | - | - |
| PUT | /api/v1/apps//secrets/ | AbilityWriteSensitive | handleSetSecret |
| GET | /api/v1/apps//secrets | AbilityRead | handleListSecrets |
| POST | /api/v1/apps//secrets//lock | AbilityWriteSensitive | handleSetSecretLock |
| GET | /api/v1/apps//git-source | AbilityRead | handleGetGitSource |
| PUT | /api/v1/apps//git-source | AbilityWriteSensitive | handleSetGitSource |
| DELETE | /api/v1/apps//git-source | AbilityWriteSensitive | handleDeleteGitSource |
| POST | /api/v1/webhooks/github/ | Public | handleGitPushWebhook |
| GET | /api/v1/apps//webhook-deliveries | AbilityRead | handleListWebhookDeliveries |
| POST | /api/v1/apps//webhook-deliveries//replay | AbilityDeploy | handleReplayWebhookDelivery |
| PUT | /api/v1/apps//preview-settings | AbilityWriteSensitive | handleSetPreviewEnabled |
| GET | /api/v1/apps//previews | AbilityRead | handleListPreviewEnvironments |
| POST | /api/v1/apps//previews//teardown | AbilityDeploy | handleTeardownPreviewEnvironment |
| POST | /api/v1/previews/sweep | AbilityDeploy | handleSweepPreviewEnvironments |
| DELETE | /api/v1/apps//secrets/ | AbilityWriteSensitive | handleDeleteSecret |
| PUT | /api/v1/apps//git-source/deploy-settings | AbilityWriteSensitive | handleSetGitDeploySettings |
| GET | /api/v1/previews | AbilityRead | handleListAllPreviews |
| POST | /api/v1/apps//previews//approve | AbilityWriteSensitive | handleApprovePreviewEnvironment |
| POST | /api/v1/apps//git-source/rotate-webhook-secret | AbilityWriteSensitive | handleRotateGitSourceWebhookSecret |
| POST | /api/v1/webhooks/slack/interactions | Public | handleSlackInteraction |
| POST | /api/v1/webhooks/discord/interactions | Public | handleDiscordInteraction |
| PUT | /api/v1/apps//git-source/build | AbilityWriteSensitive | handleSetGitSourceBuild |
| POST | /api/v1/apps//git-source/detect | AbilityDeploy | handleDetectGitSourceBuild |
| POST | /api/v1/apps//previews//extend | AbilityDeploy | handleExtendPreviewEnvironment |
| GET | /api/v1/apps//git-source/commits | AbilityDeploy | handleListGitSourceCommits |
| POST | /api/v1/apps//secrets//reveal | AbilityReadSensitive | handleRevealSecret |
| PUT | /api/v1/apps//secrets//rotation | AbilityWriteSensitive | handleSetSecretRotation |
| DELETE | /api/v1/apps//secrets//rotation | AbilityWriteSensitive | handleDeleteSecretRotation |
| POST | /api/v1/apps//secrets//rotation/rotate | AbilityWriteSensitive | handleRotateSecretNow |
| POST | /api/v1/apps//git-source/affected | AbilityRead | handleGitSourceAffected |
| GET | /api/v1/apps//previews//env | AbilityRead | handleListPRPreviewEnv |
| POST | /api/v1/apps//previews//env | AbilityWriteSensitive | handleSetPRPreviewEnv |
| DELETE | /api/v1/apps//previews//env/ | AbilityWrite | handleDeletePRPreviewEnv |
## Telemetry
Endpoints for:
* Metrics queries with filtering and aggregation
* Log retrieval, search, and live streaming
* Resource usage ranking and monitoring
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/apps//metrics | AbilityRead | handleQueryMetrics |
| GET | /api/v1/apps//logs | AbilityRead | handleQueryLogs |
| GET | /api/v1/apps/resource-usage | AbilityRead | handleAppResourceUsage |
| GET | /api/v1/apps//logs/stream | AbilityRead | handleLiveLogStream |
| GET | /api/v1/apps//logs/download | AbilityRead | handleDownloadLogs |
| GET | /api/v1/apps//logs/signatures | AbilityRead | handleLogSignatures |
## Alerts / Scheduled Tasks / Feature Flags / Notification Channels
Endpoints for:
* Alert rule creation and management
* Scheduled task configuration and execution
* [Feature flag](/feature-flags) configuration and evaluation
* Notification channel management and delivery tracking
| Method | Path | Ability | Handler |
| - | - | - | - |
| POST | /api/v1/apps//alerts | AbilityWrite | handleCreateAlertRule |
| GET | /api/v1/apps//alerts | AbilityRead | handleListAlertRules |
| PUT | /api/v1/apps//alerts/ | AbilityWrite | handleUpdateAlertRule |
| DELETE | /api/v1/apps//alerts/ | AbilityWrite | handleDeleteAlertRule |
| POST | /api/v1/apps//flags | AbilityWrite | handleCreateFeatureFlag |
| GET | /api/v1/apps//flags | AbilityRead | handleListFeatureFlags |
| GET | /api/v1/apps//flags/ | AbilityRead | handleGetFeatureFlag |
| PUT | /api/v1/apps//flags/ | AbilityWrite | handleUpdateFeatureFlag |
| DELETE | /api/v1/apps//flags/ | AbilityWrite | handleDeleteFeatureFlag |
| GET | /api/v1/flags/evaluate/ | AbilityRead | handleEvaluateFeatureFlag |
| POST | /api/v1/apps//deploy-notify-targets | AbilityWrite | handleCreateDeployNotifyTarget |
| GET | /api/v1/apps//deploy-notify-targets | AbilityRead | handleListDeployNotifyTargets |
| DELETE | /api/v1/apps//deploy-notify-targets/ | AbilityWrite | handleDeleteDeployNotifyTarget |
| GET | /api/v1/notification-channels | AbilityRead | handleListNotificationChannels |
| POST | /api/v1/notification-channels | AbilityWrite | handleCreateNotificationChannel |
| PUT | /api/v1/notification-channels/ | AbilityWrite | handleUpdateNotificationChannel |
| DELETE | /api/v1/notification-channels/ | AbilityWrite | handleDeleteNotificationChannel |
| POST | /api/v1/notification-channels/test | AbilityWrite | handleTestNotificationChannel |
| POST | /api/v1/notification-channels//test | AbilityWrite | handleTestExistingNotificationChannel |
| GET | /api/v1/notification-channels//deliveries | AbilityRead | handleListNotificationDeliveries |
| POST | /api/v1/prometheus/read | AbilityRead | handlePrometheusRead |
| POST | /api/v1/apps//alerts//silence | AbilityWrite | handleSilenceAlertRule |
## Databases CRUD / Engines / Resources
Endpoints for:
* Database creation, retrieval, and deletion
* Engine registry and telemetry
* Resource recommendations and node placement
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/database-engines | AbilityRead | handleListDatabaseEngines |
| GET | /api/v1/databases | AbilityRead | handleListDatabases |
| POST | /api/v1/databases | AbilityWrite | handleCreateDatabase |
| GET | /api/v1/databases/ | AbilityRead | handleGetDatabase |
| DELETE | /api/v1/databases/ | AbilityWrite | handleDeleteDatabase |
| GET | /api/v1/databases//status | AbilityRead | handleDatabaseStatus |
| GET | /api/v1/databases//metrics | AbilityRead | handleQueryDatabaseMetrics |
| GET | /api/v1/databases//logs | AbilityRead | handleQueryDatabaseLogs |
| GET | /api/v1/databases//logs/stream | AbilityRead | handleLiveDatabaseLogStream |
| GET | /api/v1/databases//resource-recommendation | AbilityRead | handleDatabaseResourceRecommendation |
| PUT | /api/v1/databases//node | AbilityRoot | handleSetDatabaseNode |
| GET | /api/v1/databases//slow-queries | AbilityRead | handleQueryDatabaseSlowQueries |
| POST | /api/v1/databases//pitr | AbilityWriteSensitive | handleEnablePITR |
| DELETE | /api/v1/databases//pitr | AbilityWriteSensitive | handleDisablePITR |
| GET | /api/v1/databases//pitr | AbilityRead | handleGetPITRStatus |
| POST | /api/v1/databases//base-backups | AbilityWriteSensitive | handleTriggerBaseBackup |
| GET | /api/v1/databases//base-backups | AbilityRead | handleListBaseBackupHistory |
| POST | /api/v1/databases//pitr-restore | AbilityRoot | handleTriggerPITRRestore |
| GET | /api/v1/databases//pitr-restores | AbilityRead | handleListPITRRestoreHistory |
| PUT | /api/v1/databases//version | AbilityWriteSensitive | handleSetDatabaseVersion |
| POST | /api/v1/databases//major-upgrade | AbilityRoot | handleMajorUpgrade |
| GET | /api/v1/databases//major-upgrades | AbilityRead | handleListMajorUpgrades |
| POST | /api/v1/databases//major-upgrades//rollback | AbilityRoot | handleRollbackMajorUpgrade |
| DELETE | /api/v1/databases//major-upgrades//snapshot | AbilityRoot | handleDiscardMajorUpgradeSnapshot |
| PUT | /api/v1/databases//environment | AbilityWrite | handleSetDatabaseEnvironment |
| GET | /api/v1/databases//schema | AbilityReadSensitive | handleGetDatabaseSchema |
| GET | /api/v1/databases//tables///rows | AbilityReadSensitive | handleGetDatabaseTableRows |
| POST | /api/v1/databases//query | AbilityReadSensitive | handleDatabaseQuery |
| POST | /api/v1/databases//query/write | AbilityRoot | handleDatabaseQueryWrite |
| POST | /api/v1/databases//explain | AbilityReadSensitive | handleDatabaseExplain |
| GET | /api/v1/databases//keys | AbilityReadSensitive | handleScanDatabaseKeys |
| GET | /api/v1/databases//key | AbilityReadSensitive | handleGetDatabaseKey |
| GET | /api/v1/databases//query-history | AbilityReadSensitive | handleListDatabaseQueryHistory |
| DELETE | /api/v1/databases//query-history | AbilityReadSensitive | handleClearDatabaseQueryHistory |
| GET | /api/v1/databases//saved-queries | AbilityReadSensitive | handleListDatabaseSavedQueries |
| POST | /api/v1/databases//saved-queries | AbilityReadSensitive | handleSaveDatabaseQuery |
| DELETE | /api/v1/databases//saved-queries/ | AbilityReadSensitive | handleDeleteDatabaseSavedQuery |
| GET | /api/v1/databases//users | AbilityReadSensitive | handleListDatabaseUsers |
| POST | /api/v1/databases//users | AbilityRoot | handleCreateDatabaseUser |
| POST | /api/v1/databases//users//rotate | AbilityRoot | handleRotateDatabaseUser |
| POST | /api/v1/databases//users//disable | AbilityRoot | handleDisableDatabaseUser |
| POST | /api/v1/databases//users//enable | AbilityRoot | handleEnableDatabaseUser |
| DELETE | /api/v1/databases//users/ | AbilityRoot | handleDeleteDatabaseUser |
| GET | /api/v1/databases//access/temp | AbilityReadSensitive | handleListDatabaseTempCredentials |
| POST | /api/v1/databases//access/temp | AbilityRoot | handleIssueDatabaseTempCredential |
| DELETE | /api/v1/databases//access/temp/ | AbilityRoot | handleRevokeDatabaseTempCredential |
| GET | /api/v1/databases//access/principals | AbilityReadSensitive | handleDatabaseWhoCanAccess |
| POST | /api/v1/databases//access/grants | AbilityRoot | handleGrantDatabaseAccess |
| DELETE | /api/v1/databases//access/grants/// | AbilityRoot | handleRevokeDatabaseGrant |
| GET | /api/v1/databases//network | AbilityRead | handleGetDatabaseNetwork |
| POST | /api/v1/databases//network/rules/preview | AbilityRoot | handlePreviewDatabaseRules |
| PUT | /api/v1/databases//network/rules | AbilityRoot | handleApplyDatabaseRules |
| DELETE | /api/v1/databases//network/rules | AbilityRoot | handleRemoveDatabaseRules |
| POST | /api/v1/databases//network/make-private | AbilityRoot | handleMakeDatabasePrivate |
| PUT | /api/v1/databases//network/scope | AbilityRoot | handleSetDatabaseScope |
| PUT | /api/v1/databases//network/tls | AbilityRoot | handleSetDatabaseTLS |
| GET | /api/v1/databases//tables///structure | AbilityReadSensitive | handleGetDatabaseTableStructure |
| GET | /api/v1/databases/upgrade-summary | AbilityRead | handleDatabaseUpgradeSummary |
| GET | /api/v1/databases//upgrades | AbilityRead | handleGetDatabaseUpgrades |
| PUT | /api/v1/databases//upgrade-policy | AbilityWriteSensitive | handlePutDatabaseUpgradePolicy |
| POST | /api/v1/databases//upgrade-now | AbilityWriteSensitive | handleDatabaseUpgradeNow |
| POST | /api/v1/databases//move | AbilityRoot | handleStartMove |
| GET | /api/v1/databases//placement | AbilityRead | handleGetPlacement |
| PUT | /api/v1/databases//placement | AbilityRoot | handleSetPlacement |
| GET | /api/v1/databases//engine-stats | AbilityReadSensitive | handleGetDatabaseEngineStats |
## Projects / Organizations / Environments
Endpoints for:
* Project and organization management
* Environment definitions and configuration
* Environment variable and configuration inheritance
* Bulk lifecycle operations (restart, stop, start)
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/projects | AbilityRead | handleListProjects |
| POST | /api/v1/projects | AbilityWrite | handleCreateProject |
| GET | /api/v1/projects/ | AbilityRead | handleGetProject |
| DELETE | /api/v1/projects/ | AbilityWrite | handleDeleteProject |
| POST | /api/v1/projects//restart | AbilityDeploy | handleRestartProject |
| POST | /api/v1/projects//stop | AbilityDeploy | handleStopProject |
| POST | /api/v1/projects//start | AbilityDeploy | handleStartProject |
| GET | /api/v1/organizations | AbilityRead | handleListOrganizations |
| POST | /api/v1/organizations | AbilityWrite | handleCreateOrganization |
| GET | /api/v1/organizations/ | AbilityRead | handleGetOrganization |
| DELETE | /api/v1/organizations/ | AbilityWrite | handleDeleteOrganization |
| PUT | /api/v1/projects//organization | AbilityWrite | handleSetProjectOrganization |
| GET | /api/v1/organizations//env | AbilityRead | handleGetOrganizationEnv |
| PUT | /api/v1/organizations//env | AbilityWrite | handleSetOrganizationEnv |
| GET | /api/v1/projects//environments | AbilityRead | handleListEnvironments |
| POST | /api/v1/projects//environments | AbilityWrite | handleCreateEnvironment |
| PATCH | /api/v1/environments/ | AbilityWrite | handleUpdateEnvironment |
| DELETE | /api/v1/environments/ | AbilityWrite | handleDeleteEnvironment |
| PUT | /api/v1/apps//environment | AbilityWrite | handleSetAppEnvironment |
| GET | /api/v1/environments//clone/preview | AbilityRead | handleEnvironmentClonePreview |
| POST | /api/v1/environments//clone | AbilityDeploy | handleEnvironmentClone |
| GET | /api/v1/environments//env | AbilityRead | handleGetEnvironmentEnv |
| PUT | /api/v1/environments//env | AbilityWrite | handleSetEnvironmentEnv |
| GET | /api/v1/projects//env | AbilityRead | handleGetProjectEnv |
| PUT | /api/v1/projects//env | AbilityWrite | handleSetProjectEnv |
| PUT | /api/v1/apps//project | AbilityWrite | handleSetAppProject |
| PUT | /api/v1/databases//project | AbilityWrite | handleSetDatabaseProject |
| PUT | /api/v1/databases//resources | AbilityWrite | handleSetDatabaseResources |
| GET | /api/v1/organizations//env/all | AbilityRead | handleListOrganizationEnvAll |
| GET | /api/v1/organizations//env/secrets | AbilityRead | handleListOrganizationEnvSecretKeys |
| PUT | /api/v1/organizations//env/secrets/ | AbilityWrite | handleSetOrganizationEnvSecret |
| DELETE | /api/v1/organizations//env/secrets/ | AbilityWrite | handleDeleteOrganizationEnvSecret |
| GET | /api/v1/environments//env/all | AbilityRead | handleListEnvironmentEnvAll |
| GET | /api/v1/environments//env/secrets | AbilityRead | handleListEnvironmentEnvSecretKeys |
| PUT | /api/v1/environments//env/secrets/ | AbilityWrite | handleSetEnvironmentEnvSecret |
| DELETE | /api/v1/environments//env/secrets/ | AbilityWrite | handleDeleteEnvironmentEnvSecret |
| GET | /api/v1/projects//env/all | AbilityRead | handleListProjectEnvAll |
| GET | /api/v1/projects//env/secrets | AbilityRead | handleListProjectEnvSecretKeys |
| PUT | /api/v1/projects//env/secrets/ | AbilityWrite | handleSetProjectEnvSecret |
| DELETE | /api/v1/projects//env/secrets/ | AbilityWrite | handleDeleteProjectEnvSecret |
| GET | /api/v1/projects//environments/compare | AbilityRead | handleCompareEnvironmentEnv |
| GET | /api/v1/projects//topology | AbilityRead | handleGetProjectTopology |
| GET | /api/v1/environments | AbilityRead | handleListAllEnvironments |
| POST | /api/v1/environments | AbilityWrite | handleCreateGlobalEnvironment |
| GET | /api/v1/projects//quota | AbilityRead | handleGetQuota |
| PUT | /api/v1/projects//quota | AbilityWrite | handleSetQuota |
| DELETE | /api/v1/projects//quota | AbilityWrite | handleDeleteQuota |
## Nodes
Endpoints for:
* Physical infrastructure management and monitoring
* Node health and status
* Workload assignment and placement
* Cordon, drain, and lifecycle operations
* Node-level metrics and patch status
* Agent certificate re-enrollment and revocation
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/nodes | AbilityRoot | handleListNodes |
| GET | /api/v1/nodes/ | AbilityRoot | handleGetNode |
| DELETE | /api/v1/nodes/ | AbilityRoot | handleDeleteNode |
| PUT | /api/v1/nodes//workloads | AbilityRoot | handleSetNodeWorkloads |
| POST | /api/v1/nodes/join-tokens | AbilityRoot | handleCreateNodeJoinToken |
| GET | /api/v1/nodes//health | AbilityRoot | handleGetNodeHealth |
| POST | /api/v1/nodes//cordon | AbilityRoot | handleCordonNode |
| POST | /api/v1/nodes//uncordon | AbilityRoot | handleUncordonNode |
| POST | /api/v1/nodes//drain | AbilityRoot | handleDrainNode |
| GET | /api/v1/nodes//metrics | AbilityRoot | handleQueryNodeMetrics |
| GET | /api/v1/nodes//patch-status | AbilityRoot | handleGetNodePatchStatus |
| GET | /api/v1/nodes//events | AbilityRoot | handleListNodeEvents |
| GET | /api/v1/nodes//capacity-forecast | AbilityRoot | handleNodeCapacityForecast |
| POST | /api/v1/nodes//reenroll-token | AbilityRoot | handleCreateNodeReenrollToken |
| POST | /api/v1/nodes//revoke-cert | AbilityRoot | handleRevokeNodeCert |
| POST | /api/v1/nodes//mesh/rotate-key | AbilityRoot | handleRotateNodeMeshKey |
| GET | /api/v1/nodes/resource-usage | AbilityRoot | handleFleetResourceUsage |
| POST | /api/v1/nodes/provision | AbilityRoot | handleCreateNodeProvision |
| PUT | /api/v1/nodes//region | AbilityRoot | handleSetNodeRegion |
| POST | /api/v1/nodes/ssh-provision | AbilityRoot | handleCreateSSHNodeProvision |
| POST | /api/v1/nodes//mesh/rejoin | AbilityRoot | handleRejoinNodeMesh |
| GET | /api/v1/nodes//drains | AbilityRoot | handleListNodeDrains |
| GET | /api/v1/nodes//labels | AbilityRoot | handleGetNodeLabels |
| PUT | /api/v1/nodes//labels | AbilityRoot | handleSetNodeLabels |
| POST | /api/v1/nodes/ssh-provision/preflight | AbilityRoot | handleSSHProvisionPreflight |
| GET | /api/v1/nodes//security | AbilityRoot | handleGetNodeTrust |
| GET | /api/v1/nodes//security/events | AbilityRoot | handleListNodeSecurityEvents |
| POST | /api/v1/nodes//cert/renew | AbilityRoot | handleRenewNodeCert |
## Ingress / Certificates / Domains / Email / Cloudflare
Endpoints for:
* TLS certificate lifecycle and management
* ACME configuration and certificate validation
* Domain routing and DNS configuration
* Domain-level controls (basic auth, maintenance mode, redirects, custom error pages, WAF)
* Email configuration for notifications
* Cloudflare, Route 53, and Vault integrations
* External secret management
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/certificates | AbilityRead | handleListCertificates |
| GET | /api/v1/settings/ingress | AbilityRead | handleGetIngressSettings |
| PUT | /api/v1/settings/ingress | AbilityRoot | handleUpdateIngressSettings |
| GET | /api/v1/settings/ingress/check | AbilityRoot | handleCheckIngressDomain |
| GET | /api/v1/settings/dashboard-url | AbilityRead | handleGetDashboardURL |
| PUT | /api/v1/settings/dashboard-url | AbilityRoot | handleUpdateDashboardURL |
| GET | /api/v1/apps//domains//check | AbilityRead | handleCheckDomain |
| GET | /api/v1/apps//domains//auth | AbilityRead | handleGetDomainBasicAuth |
| PUT | /api/v1/apps//domains//auth | AbilityRoot | handleSetDomainBasicAuth |
| DELETE | /api/v1/apps//domains//auth | AbilityRoot | handleClearDomainBasicAuth |
| GET | /api/v1/apps//domains//maintenance | AbilityRead | handleGetDomainMaintenance |
| PUT | /api/v1/apps//domains//maintenance | AbilityDeploy | handleSetDomainMaintenance |
| DELETE | /api/v1/apps//domains//maintenance | AbilityDeploy | handleClearDomainMaintenance |
| GET | /api/v1/apps//domains//redirect | AbilityRead | handleGetDomainRedirect |
| PUT | /api/v1/apps//domains//redirect | AbilityDeploy | handleSetDomainRedirect |
| DELETE | /api/v1/apps//domains//redirect | AbilityDeploy | handleClearDomainRedirect |
| GET | /api/v1/apps//domains//tls-cert | AbilityRead | handleGetDomainTLSCert |
| PUT | /api/v1/apps//domains//tls-cert | AbilityRoot | handleSetDomainTLSCert |
| DELETE | /api/v1/apps//domains//tls-cert | AbilityRoot | handleClearDomainTLSCert |
| GET | /api/v1/apps//domains//waf | AbilityRead | handleGetDomainWAF |
| PUT | /api/v1/apps//domains//waf | AbilityDeploy | handleSetDomainWAF |
| DELETE | /api/v1/apps//domains//waf | AbilityDeploy | handleClearDomainWAF |
| GET | /api/v1/apps//domains//error-pages | AbilityRead | handleGetDomainErrorPages |
| PUT | /api/v1/apps//domains//error-pages | AbilityDeploy | handleSetDomainErrorPage |
| DELETE | /api/v1/apps//domains//error-pages | AbilityDeploy | handleClearDomainErrorPages |
| GET | /api/v1/settings/email | AbilityRead | handleGetEmailSettings |
| PUT | /api/v1/settings/email | AbilityRoot | handleUpdateEmailSettings |
| GET | /api/v1/settings/cloudflare-tunnel | AbilityRead | handleGetCloudflareTunnelSettings |
| PUT | /api/v1/settings/cloudflare-tunnel | AbilityRoot | handleUpdateCloudflareTunnelSettings |
| DELETE | /api/v1/settings/cloudflare-tunnel | AbilityRoot | handleDisconnectCloudflareTunnel |
| GET | /api/v1/settings/cloudflare-dns | AbilityRead | handleGetCloudflareDNSSettings |
| PUT | /api/v1/settings/cloudflare-dns | AbilityRoot | handleUpdateCloudflareDNSSettings |
| DELETE | /api/v1/settings/cloudflare-dns | AbilityRoot | handleDisconnectCloudflareDNS |
| GET | /api/v1/settings/route53-dns | AbilityRead | handleGetRoute53DNSSettings |
| PUT | /api/v1/settings/route53-dns | AbilityRoot | handleUpdateRoute53DNSSettings |
| DELETE | /api/v1/settings/route53-dns | AbilityRoot | handleDisconnectRoute53DNS |
| GET | /api/v1/settings/vault | AbilityRead | handleGetVaultSettings |
| PUT | /api/v1/settings/vault | AbilityRoot | handleUpdateVaultSettings |
| DELETE | /api/v1/settings/vault | AbilityRoot | handleDisconnectVault |
| PUT | /api/v1/apps//vault-env/ | AbilityWrite | handleSetAppVaultEnv |
| DELETE | /api/v1/apps//vault-env/ | AbilityWrite | handleClearAppVaultEnv |
| PUT | /api/v1/apps//preview-env/ | AbilityWrite | handleSetAppPreviewEnvOverride |
| DELETE | /api/v1/apps//preview-env/ | AbilityWrite | handleClearAppPreviewEnvOverride |
| GET | /api/v1/apps//branch-env | AbilityRead | handleListAppBranchEnv |
| POST | /api/v1/apps//branch-env | AbilityWriteSensitive | handleSetAppBranchEnv |
| DELETE | /api/v1/apps//branch-env/ | AbilityWrite | handleDeleteAppBranchEnv |
| PATCH | /api/v1/apps//domains | AbilityWrite | handleEditAppDomains |
| POST | /api/v1/apps//domains//cert/renew | AbilityRoot | handleRenewDomainCertificate |
| GET | /api/v1/apps//domains//dns-records | AbilityRead | handleListDNSRecords |
| POST | /api/v1/apps//domains//dns-records | AbilityRoot | handleCreateDNSRecord |
| PUT | /api/v1/apps//domains//dns-records | AbilityRoot | handleUpdateDNSRecord |
| DELETE | /api/v1/apps//domains//dns-records | AbilityRoot | handleDeleteDNSRecord |
| POST | /api/v1/settings/email/test | AbilityWrite | handleTestEmail |
| GET | /api/v1/settings/ingress/https | AbilityRead | handleGetHTTPSStatus |
| POST | /api/v1/settings/ingress/https | AbilityRoot | handleEnableHTTPS |
| GET | /api/v1/apps//domains//search-visibility | AbilityRead | handleGetDomainSearchVisibility |
| PUT | /api/v1/apps//domains//search-visibility | AbilityDeploy | handleSetDomainSearchVisibility |
| GET | /api/v1/apps//domains//headers | AbilityRead | handleGetDomainPolicy |
| PUT | /api/v1/apps//domains//headers | AbilityDeploy | handleSetDomainPolicy |
| DELETE | /api/v1/apps//domains//headers | AbilityDeploy | handleDeleteDomainPolicy |
| GET | /api/v1/apps//domains//forwarders | AbilityRead | handleGetDomainPolicy |
| PUT | /api/v1/apps//domains//forwarders | AbilityDeploy | handleSetDomainPolicy |
| DELETE | /api/v1/apps//domains//forwarders | AbilityDeploy | handleDeleteDomainPolicy |
| GET | /api/v1/apps//domains//geo | AbilityRead | handleGetDomainPolicy |
| PUT | /api/v1/apps//domains//geo | AbilityDeploy | handleSetDomainPolicy |
| DELETE | /api/v1/apps//domains//geo | AbilityDeploy | handleDeleteDomainPolicy |
| GET | /api/v1/apps//domains//cache | AbilityRead | handleGetDomainPolicy |
| PUT | /api/v1/apps//domains//cache | AbilityDeploy | handleSetDomainPolicy |
| DELETE | /api/v1/apps//domains//cache | AbilityDeploy | handleDeleteDomainPolicy |
| POST | /api/v1/apps//domains//cache/purge | AbilityDeploy | handlePurgeDomainCache |
| GET | /api/v1/apps//domains//cache/stats | AbilityRead | handleGetDomainCacheStats |
| GET | /api/v1/apps//domains//policies | AbilityRead | handleGetDomainPolicies |
| POST | /api/v1/apps//domains//policies/preview | AbilityRead | handlePreviewDomainPolicies |
| GET | /api/v1/apps//domains//redirects | AbilityRead | handleGetDomainRedirects |
| PUT | /api/v1/apps//domains//redirects | AbilityDeploy | handleSetDomainRedirectSettings |
| DELETE | /api/v1/apps//domains//redirects | AbilityDeploy | handleDeleteDomainRedirectSettings |
| POST | /api/v1/apps//domains//redirects/canonical | AbilityDeploy | handleSetDomainCanonical |
| PUT | /api/v1/apps//domains//redirects/aliases | AbilityDeploy | handleSetDomainAliases |
| GET | /api/v1/apps//domains//ports | AbilityRead | handleGetDomainPorts |
| PUT | /api/v1/apps//domains//ports//restrict | AbilityWriteSensitive | handleRestrictDomainPort |
| DELETE | /api/v1/apps//domains//ports//restrict | AbilityWriteSensitive | handleUnrestrictDomainPort |
| GET | /api/v1/apps//domains//go-live | AbilityRead | handleGoLiveStatus |
| POST | /api/v1/apps//domains//go-live | AbilityRoot | handleGoLiveRun |
| POST | /api/v1/apps//domains/go-live/plan | AbilityWrite | handleGoLivePlan |
| POST | /api/v1/settings/ingress/apps-base-domain/backfill | AbilityRoot | handleBackfillBaseDomain |
| POST | /api/v1/apps//domains//doctor | AbilityRead | handleDomainDoctor |
| GET | /api/v1/apps//domains//auth-gate | AbilityRead | handleGetDomainAuthGate |
| PUT | /api/v1/apps//domains//auth-gate | AbilityRoot | handleSetDomainAuthGate |
| DELETE | /api/v1/apps//domains//auth-gate | AbilityRoot | handleClearDomainAuthGate |
| GET | /api/v1/apps//domains//auth-gate/events | AbilityReadSensitive | handleListAuthGateEvents |
| DELETE | /api/v1/apps//domains//auth-gate/sessions | AbilityRoot | handleRevokeAuthGateSessions |
## Static Sites / Backup Targets / Registry Credentials
Endpoints for:
* Static site configuration and visibility
* S3-compatible backup target management and testing
* External registry credentials for image pull authentication
* Domain enumeration
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/static-sites | AbilityRead | handleListStaticSites |
| GET | /api/v1/backup-targets | AbilityRead | handleListBackupTargets |
| POST | /api/v1/backup-targets | AbilityWriteSensitive | handleCreateBackupTarget |
| GET | /api/v1/backup-targets/ | AbilityRead | handleGetBackupTarget |
| PUT | /api/v1/backup-targets/ | AbilityWriteSensitive | handleUpdateBackupTarget |
| DELETE | /api/v1/backup-targets/ | AbilityWriteSensitive | handleDeleteBackupTarget |
| POST | /api/v1/backup-targets//test | AbilityWriteSensitive | handleTestBackupTarget |
| GET | /api/v1/registry-credentials | AbilityRead | handleListRegistryCredentials |
| POST | /api/v1/registry-credentials | AbilityWriteSensitive | handleCreateRegistryCredential |
| GET | /api/v1/registry-credentials/ | AbilityRead | handleGetRegistryCredential |
| PUT | /api/v1/registry-credentials/ | AbilityWriteSensitive | handleUpdateRegistryCredential |
| DELETE | /api/v1/registry-credentials/ | AbilityWriteSensitive | handleDeleteRegistryCredential |
| POST | /api/v1/registry-credentials//test | AbilityWriteSensitive | handleTestRegistryCredential |
| GET | /api/v1/registry-credentials//repositories | AbilityReadSensitive | handleListRegistryCredentialRepositories |
| GET | /api/v1/registry-credentials//tags | AbilityReadSensitive | handleListRegistryCredentialTags |
| GET | /api/v1/domains | AbilityRead | handleListDomains |
| GET | /api/v1/domains/automation/runs | AbilityRead | handleListAutomationRuns |
| POST | /api/v1/domains/automation/runs//undo | AbilityRoot | handleUndoAutomationRun |
| GET | /api/v1/domains//activity | AbilityRead | handleDomainActivity |
| GET | /api/v1/domains/standalone | AbilityRead | handleListStandaloneDomains |
| POST | /api/v1/domains/standalone | AbilityWriteSensitive | handleCreateStandaloneDomain |
| POST | /api/v1/domains/standalone/validate | AbilityWriteSensitive | handleValidateStandaloneDomain |
| GET | /api/v1/domains/standalone/candidates | AbilityWriteSensitive | handleStandaloneCandidates |
| PUT | /api/v1/domains/standalone/ | AbilityWriteSensitive | handleUpdateStandaloneDomain |
| DELETE | /api/v1/domains/standalone/ | AbilityWriteSensitive | handleDeleteStandaloneDomain |
| GET | /api/v1/registry-credentials//usage | AbilityRead | handleRegistryCredentialUsage |
| POST | /api/v1/registry-credentials/test | AbilityWriteSensitive | handleProbeRegistryCredential |
| DELETE | /api/v1/static-sites/ | AbilityWriteSensitive | handleDeleteStaticSite |
## Built-in Container Registry
Endpoints for:
* Registry configuration and management
* Image repository and tag browsing
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/settings/registry | AbilityRead | handleGetRegistrySettings |
| PUT | /api/v1/settings/registry | AbilityRoot | handleUpdateRegistrySettings |
| DELETE | /api/v1/settings/registry | AbilityRoot | handleDisableRegistry |
| GET | /api/v1/registry/repositories | AbilityRead | handleListRegistryRepositories |
| GET | /api/v1/registry/tags | AbilityRead | handleListRegistryTags |
| GET | /api/v1/dockerhub/search | AbilityRead | handleDockerHubSearch |
| GET | /api/v1/dockerhub/repositories///tags | AbilityRead | handleDockerHubTags |
## Git Provider Apps
Endpoints for:
* GitHub App integration and repository browsing
* GitLab App integration and project browsing
* Bitbucket App integration and repository browsing
* OAuth flow management and provider status
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/git-providers | AbilityReadSensitive | handleListGitProviders |
| GET | /api/v1/github-app | AbilityRoot | handleGetGitHubAppStatus |
| DELETE | /api/v1/github-app | AbilityRoot | handleDisconnectGitHubApp |
| PUT | /api/v1/github-app/manual | AbilityRoot | handleConnectGitHubAppManually |
| GET | /api/v1/github-app/register/preview | AbilityRoot | handleGetGitHubAppManifestPreview |
| GET | /api/v1/github-app/register/start | AbilityRoot | handleStartGitHubAppRegistration |
| GET | /api/v1/github-app/callback | AbilityRoot | handleGitHubAppCallback |
| GET | /api/v1/github-app/installed | AbilityRoot | handleGitHubAppInstalled |
| GET | /api/v1/github-app/repos | AbilityReadSensitive | handleListGitHubAppRepos |
| GET | /api/v1/github-app/repos///branches | AbilityReadSensitive | handleListGitHubAppBranches |
| POST | /api/v1/github-app/repos///use-as-source | AbilityWriteSensitive | handleUseGitHubRepoAsSource |
| GET | /api/v1/gitlab-app | AbilityRoot | handleGetGitLabAppStatus |
| PUT | /api/v1/gitlab-app | AbilityRoot | handleConnectGitLabApp |
| DELETE | /api/v1/gitlab-app | AbilityRoot | handleDisconnectGitLabApp |
| GET | /api/v1/gitlab-app/connect | AbilityRoot | handleStartGitLabAppConnect |
| GET | /api/v1/gitlab-app/callback | AbilityRoot | handleGitLabAppCallback |
| GET | /api/v1/gitlab-app/projects | AbilityReadSensitive | handleListGitLabAppProjects |
| GET | /api/v1/gitlab-app/projects//branches | AbilityReadSensitive | handleListGitLabAppBranches |
| POST | /api/v1/gitlab-app/projects//use-as-source | AbilityWriteSensitive | handleUseGitLabProjectAsSource |
| GET | /api/v1/bitbucket-app | AbilityRoot | handleGetBitbucketAppStatus |
| PUT | /api/v1/bitbucket-app | AbilityRoot | handleConnectBitbucketApp |
| DELETE | /api/v1/bitbucket-app | AbilityRoot | handleDisconnectBitbucketApp |
| GET | /api/v1/bitbucket-app/connect | AbilityRoot | handleStartBitbucketAppConnect |
| GET | /api/v1/bitbucket-app/callback | AbilityRoot | handleBitbucketAppCallback |
| GET | /api/v1/bitbucket-app/repos | AbilityReadSensitive | handleListBitbucketAppRepos |
| GET | /api/v1/bitbucket-app/repos///branches | AbilityReadSensitive | handleListBitbucketAppBranches |
| POST | /api/v1/bitbucket-app/repos///use-as-source | AbilityWriteSensitive | handleUseBitbucketRepoAsSource |
| GET | /api/v1/github-app/installations | AbilityRoot | handleListGitHubAppInstallations |
| DELETE | /api/v1/github-app/installations/ | AbilityRoot | handleDeleteGitHubAppInstallation |
## Database Backups / Restore / Clone Restore
Endpoints for:
* Backup history and manual triggers
* Backup verification and scheduled retention
* Database control (stop, start, public access)
* Restore and clone-restore operations
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/backups | AbilityRead | handleListAllBackups |
| POST | /api/v1/databases//backups | AbilityWriteSensitive | handleTriggerBackup |
| GET | /api/v1/databases//backups | AbilityRead | handleListBackupHistory |
| GET | /api/v1/databases//backups//download | AbilityReadSensitive | handleDownloadBackup |
| POST | /api/v1/databases//backups//verify | AbilityWriteSensitive | handleVerifyBackup |
| GET | /api/v1/databases//backups//verifications | AbilityRead | handleListBackupVerifications |
| PUT | /api/v1/databases//backup-schedule | AbilityWriteSensitive | handleSetBackupSchedule |
| DELETE | /api/v1/databases//backup-schedule | AbilityWriteSensitive | handleClearBackupSchedule |
| PUT | /api/v1/databases//public-access | AbilityWriteSensitive | handleSetDatabasePublicAccess |
| DELETE | /api/v1/databases//public-access | AbilityWriteSensitive | handleClearDatabasePublicAccess |
| POST | /api/v1/databases//stop | AbilityWriteSensitive | handleStopDatabase |
| POST | /api/v1/databases//start | AbilityWriteSensitive | handleStartDatabase |
| POST | /api/v1/databases//restore | AbilityRoot | handleTriggerRestore |
| GET | /api/v1/databases//restores | AbilityRead | handleListRestoreHistory |
| POST | /api/v1/databases//restore-as-new | AbilityWriteSensitive | handleCloneRestore |
| GET | /api/v1/databases//clone-restores | AbilityRead | handleListCloneRestores |
| DELETE | /api/v1/databases//backups/ | AbilityWriteSensitive | handleDeleteBackup |
| GET | /api/v1/backups/health | AbilityRead | handleBackupHealth |
| GET | /api/v1/backups/drills | AbilityRead | handleListBackupDrills |
| GET | /api/v1/backups/drills/ | AbilityRead | handleGetBackupDrill |
| POST | /api/v1/backups/drills | AbilityWriteSensitive | handleStartBackupDrill |
| POST | /api/v1/backups/protection/refresh | AbilityWriteSensitive | handleRefreshBackupProtection |
| GET | /api/v1/backups/coverage | AbilityRead | handleBackupCoverage |
| POST | /api/v1/backups/protect-all | AbilityWriteSensitive | handleProtectAll |
| POST | /api/v1/backups/encryption-key/ack | AbilityWriteSensitive | handleAckBackupEncryptionKey |
## App Volume Backups / Restore
Endpoints for:
* Named volume backup history and scheduling
* Backup verification and retention management
* Volume restore operations (in-place and non-destructive clone)
| Method | Path | Ability | Handler |
| - | - | - | - |
| POST | /api/v1/apps//volumes//backups | AbilityWriteSensitive | handleTriggerVolumeBackup |
| GET | /api/v1/apps//volumes//backups | AbilityRead | handleListVolumeBackupHistory |
| GET | /api/v1/apps//volumes//backups//download | AbilityReadSensitive | handleDownloadVolumeBackup |
| POST | /api/v1/apps//volumes//backups//verify | AbilityWriteSensitive | handleVerifyVolumeBackup |
| GET | /api/v1/apps//volumes//backups//verifications | AbilityRead | handleListVolumeBackupVerifications |
| GET | /api/v1/apps//volumes//backup-schedule | AbilityRead | handleGetVolumeBackupSchedule |
| PUT | /api/v1/apps//volumes//backup-schedule | AbilityWriteSensitive | handleSetVolumeBackupSchedule |
| DELETE | /api/v1/apps//volumes//backup-schedule | AbilityWriteSensitive | handleClearVolumeBackupSchedule |
| POST | /api/v1/apps//volumes//restore | AbilityRoot | handleTriggerVolumeRestore |
| GET | /api/v1/apps//volumes//restores | AbilityRead | handleListVolumeRestoreHistory |
| POST | /api/v1/apps//volumes//restore-as-new | AbilityWriteSensitive | handleVolumeCloneRestore |
| GET | /api/v1/apps//volumes//clone-restores | AbilityRead | handleListVolumeCloneRestores |
| DELETE | /api/v1/apps//volumes//backups/ | AbilityWriteSensitive | handleDeleteVolumeBackup |
| PUT | /api/v1/apps//volumes | AbilityWrite | handleSetAppVolumes |
| GET | /api/v1/apps//volumes//backup-policy | AbilityRead | handleGetVolumeBackupPolicy |
| PUT | /api/v1/apps//volumes//backup-policy | AbilityWriteSensitive | handleSetVolumeBackupPolicy |
| POST | /api/v1/apps//volumes//restore-to | AbilityWriteSensitive | handleVolumeRestoreTo |
## App Storage / Database Attach
Endpoints for:
* [Object storage](/object-storage) bucket attachment and configuration
* Database connectivity for applications
| Method | Path | Ability | Handler |
| - | - | - | - |
| PUT | /api/v1/apps//storage | AbilityWriteSensitive | handleSetAppStorage |
| DELETE | /api/v1/apps//storage | AbilityWriteSensitive | handleClearAppStorage |
| PUT | /api/v1/apps//database | AbilityWrite | handleSetAppDatabase |
| DELETE | /api/v1/apps//database | AbilityWrite | handleClearAppDatabase |
| GET | /api/v1/storage-env-keys | AbilityRead | handleListStorageEnvKeys |
## Audit Log / Log Drain
Endpoints for:
* Fleet-wide audit trail and log management
* External log sink configuration per application
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/audit-log | AbilityRoot | handleListAuditLog |
| POST | /api/v1/audit-log/purge | AbilityRoot | handlePurgeAuditLog |
| GET | /api/v1/apps//log-drain | AbilityRead | handleGetAppLogDrain |
| PUT | /api/v1/apps//log-drain | AbilityWriteSensitive | handleSetAppLogDrain |
| DELETE | /api/v1/apps//log-drain | AbilityWriteSensitive | handleClearAppLogDrain |
| GET | /api/v1/audit-log/export | AbilityRoot | handleExportAuditLog |
`GET /api/v1/audit-log` accepts optional query params: `limit`, `before` (RFC3339 cursor), `path`, `method`, `client_kind` (exact match), `q` (case-insensitive substring across actor name, ability, method, path and remote address), `status=failed` (only `status_code >= 400`), and `format=csv`. All filters combine with AND and the CSV export honors them.
## AI Models / GPUs
AI model resources on GPU nodes and the GPU node snapshots they schedule against. Details: [AI models](/ai-models).
| Method | Path | Ability | Handler |
| - | - | - | - |
| GET | /api/v1/models | AbilityRead | handleListModels |
| POST | /api/v1/models | AbilityWriteSensitive | handleCreateModel |
| GET | /api/v1/models/ | AbilityRead | handleGetModel |
| DELETE | /api/v1/models/ | AbilityWrite | handleDeleteModel |
| POST | /api/v1/models//restart | AbilityWrite | handleRestartModel |
| POST | /api/v1/models//api-key | AbilityWriteSensitive | handleRotateModelAPIKey |
| PUT | /api/v1/models//hf-token | AbilityWriteSensitive | handleSetModelHFToken |
| GET | /api/v1/models//logs | AbilityRead | handleQueryModelLogs |
| GET | /api/v1/models//logs/stream | AbilityRead | handleLiveModelLogStream |
| GET | /api/v1/gpus | AbilityRead | handleListGPUNodes |
| GET | /api/v1/models//keys | AbilityRead | handleListModelKeys |
| POST | /api/v1/models//keys | AbilityWriteSensitive | handleCreateModelKey |
| DELETE | /api/v1/models//keys/ | AbilityWrite | handleRevokeModelKey |
| POST | /api/v1/models//keys//rotate | AbilityWriteSensitive | handleRotateModelKey |
| GET | /api/v1/models//usage | AbilityRead | handleModelUsage |
| POST | /api/v1/models/preflight | AbilityRead | handleModelPreflight |
| GET | /api/v1/models//engine-metrics | AbilityRead | handleModelEngineMetrics |
| GET | /api/v1/models//fit | AbilityRead | handleModelFit |
| POST | /api/v1/models/fit | AbilityRead | handleModelFitCheck |
| PUT | /api/v1/models//residency | AbilityWrite | handleSetModelResidency |
| POST | /api/v1/models//wake | AbilityWrite | handleWakeModel |
| POST | /api/v1/models//sleep | AbilityWrite | handleSleepModel |
| PUT | /api/v1/models//swap-group | AbilityWrite | handleSetModelSwapGroup |
| POST | /api/v1/models//benchmark | AbilityWrite | handleModelBenchmark |
## Other
Routes that do not fit an existing group.
| Method | Path | Ability | Handler |
| - | - | - | - |
| POST | /api/v1/build/detect | AbilityDeploy | handleDetectFramework |
| POST | /api/v1/pipelines/validate | AbilityRead | handleValidatePipeline |
| GET | /api/v1/pipelines/schema | AbilityRead | handlePipelineSchema |
| POST | /api/v1/tags | AbilityWrite | handleCreateTag |
| GET | /api/v1/tags | AbilityRead | handleListTags |
| DELETE | /api/v1/tags/ | AbilityWrite | handleDeleteTag |
| GET | /api/v1/tags//apps | AbilityRead | handleListAppsByTag |
| GET | /api/v1/integrations | AbilityRead | handleListIntegrationCatalog |
| GET | /api/v1/deploy-approvals | AbilityRead | handleListDeployApprovals |
| GET | /api/v1/deploy-approvals/ | AbilityRead | handleGetDeployApproval |
| POST | /api/v1/deploy-approvals//approve | AbilityDeploy | handleApproveDeployApproval |
| POST | /api/v1/deploy-approvals//reject | AbilityDeploy | handleRejectDeployApproval |
| GET | /api/v1/mesh | AbilityRoot | handleGetMeshStatus |
| GET | /api/v1/gitea-app | AbilityRoot | handleGetGiteaAppStatus |
| PUT | /api/v1/gitea-app | AbilityRoot | handleConnectGiteaApp |
| DELETE | /api/v1/gitea-app | AbilityRoot | handleDisconnectGiteaApp |
| GET | /api/v1/gitea-app/connect | AbilityRoot | handleStartGiteaAppConnect |
| GET | /api/v1/gitea-app/callback | AbilityRoot | handleGiteaAppCallback |
| GET | /api/v1/gitea-app/repos | AbilityReadSensitive | handleListGiteaAppRepos |
| GET | /api/v1/gitea-app/repos///branches | AbilityReadSensitive | handleListGiteaAppBranches |
| POST | /api/v1/gitea-app/repos///use-as-source | AbilityWriteSensitive | handleUseGiteaRepoAsSource |
| POST | /api/v1/ai/sessions | AbilityRoot | handleCreateAIChatSession |
| GET | /api/v1/ai/sessions | AbilityRoot | handleListAIChatSessions |
| GET | /api/v1/ai/sessions/ | AbilityRoot | handleGetAIChatSession |
| DELETE | /api/v1/ai/sessions/ | AbilityRoot | handleDeleteAIChatSession |
| POST | /api/v1/ai/sessions//messages | AbilityRoot | handleCreateAIChatMessage |
| POST | /api/v1/ai/sessions//confirmations/ | AbilityRoot | handleResolveAIChatConfirmation |
| GET | /api/v1/storage/providers | AbilityRead | handleListStorageProviders |
| GET | /api/v1/storage/destinations | AbilityRead | handleListStorageDestinations |
| POST | /api/v1/storage/destinations | AbilityWriteSensitive | handleCreateStorageDestination |
| GET | /api/v1/storage/destinations/ | AbilityRead | handleGetStorageDestination |
| PUT | /api/v1/storage/destinations/ | AbilityWriteSensitive | handleUpdateStorageDestination |
| DELETE | /api/v1/storage/destinations/ | AbilityWriteSensitive | handleDeleteStorageDestination |
| POST | /api/v1/storage/destinations//test | AbilityWriteSensitive | handleTestStorageDestination |
| GET | /api/v1/log-archive/policies | AbilityRead | handleListLogArchivePolicies |
| PUT | /api/v1/log-archive/policy | AbilityWrite | handleSetLogArchivePolicy |
| DELETE | /api/v1/log-archive/policy | AbilityWrite | handleDeleteLogArchivePolicy |
| POST | /api/v1/log-archive/dump | AbilityWrite | handleLogArchiveDump |
| GET | /api/v1/log-archive/runs | AbilityRead | handleListLogArchiveRuns |
| GET | /api/v1/log-archive/objects | AbilityRead | handleListLogArchiveObjects |
| GET | /api/v1/log-archive/objects/download | AbilityRead | handleDownloadLogArchiveObject |
| GET | /api/v1/build-cache | AbilityRead | handleListBuildCache |
| PUT | /api/v1/build-cache | AbilityWrite | handleSetBuildCache |
| DELETE | /api/v1/build-cache | AbilityWrite | handleDeleteBuildCache |
| GET | /api/v1/build-cache/stats | AbilityRead | handleBuildCacheStats |
| POST | /api/v1/build-cache/clear | AbilityWrite | handleClearBuildCache |
| GET | /api/v1/pipeline-runs | AbilityRead | handleListAllPipelineRuns |
| GET | /api/v1/pipelines/summary | AbilityRead | handleGetPipelineSummary |
| GET | /api/v1/loadbalancers | AbilityRead | handleListLoadBalancers |
| GET | /api/v1/alert-silences | AbilityRead | handleListAlertSilences |
| POST | /api/v1/alert-silences | AbilityWrite | handleCreateAlertSilence |
| DELETE | /api/v1/alert-silences/ | AbilityWrite | handleExpireAlertSilence |
| GET | /api/v1/alert-maintenance-windows | AbilityRead | handleListMaintenanceWindows |
| POST | /api/v1/alert-maintenance-windows | AbilityWrite | handleCreateMaintenanceWindow |
| PUT | /api/v1/alert-maintenance-windows/ | AbilityWrite | handleUpdateMaintenanceWindow |
| DELETE | /api/v1/alert-maintenance-windows/ | AbilityWrite | handleDeleteMaintenanceWindow |
| GET | /api/v1/alert-history | AbilityRead | handleListAlertHistory |
| GET | /public/status | Public | handlePublicStatusHTML |
| GET | /public/status.json | Public | handlePublicStatusJSON |
| GET | /public/status.rss | Public | handlePublicStatusRSS |
| GET | /api/v1/status-page | AbilityRead | handleGetStatusPage |
| PUT | /api/v1/status-page | AbilityWrite | handlePutStatusPage |
| GET | /api/v1/status-page/preview | AbilityRead | handleStatusPagePreview |
| GET | /api/v1/status-page/components | AbilityRead | handleListStatusComponents |
| POST | /api/v1/status-page/components | AbilityWrite | handleCreateStatusComponent |
| PUT | /api/v1/status-page/components/ | AbilityWrite | handleUpdateStatusComponent |
| DELETE | /api/v1/status-page/components/ | AbilityWrite | handleDeleteStatusComponent |
| GET | /api/v1/status-page/incidents | AbilityRead | handleListStatusIncidents |
| POST | /api/v1/status-page/incidents | AbilityWrite | handleCreateStatusIncident |
| POST | /api/v1/status-page/incidents//updates | AbilityWrite | handlePostStatusIncidentUpdate |
| DELETE | /api/v1/status-page/incidents/ | AbilityWrite | handleDeleteStatusIncident |
| GET | /api/v1/apps-summary | AbilityRead | handleAppsSummary |
| POST | /api/v1/preflight | AbilityWrite | handlePreflightNew |
| POST | /api/v1/imports/plan | AbilityWrite | handleImportPlan |
| POST | /api/v1/apply/plan | AbilityRead | handleIaCPlan |
| POST | /api/v1/apply | AbilityWrite | handleIaCApply |
| GET | /api/v1/export | AbilityRead | handleIaCExport |
| GET | /api/v1/apply/schema | AbilityRead | handleIaCSchema |
| POST | /api/v1/imports/platform/discover | AbilityWriteSensitive | handleDiscoverPlatformImport |
| POST | /api/v1/imports/platform/apply | AbilityWriteSensitive | handleApplyPlatformImport |
| GET | /api/v1/deployments | AbilityRead | handleListDeployments |
| GET | /api/v1/deployments/summary | AbilityRead | handleDeploymentsSummary |
| GET | /api/v1/deployments/stream | AbilityRead | handleDeploymentsStream |
| GET | /api/v1/model-cache | AbilityRead | handleListModelCache |
| POST | /api/v1/model-cache/prune | AbilityRoot | handlePruneModelCache |
| POST | /api/v1/pipelines/filters | AbilityRead | handlePipelineFilters |
| GET | /api/v1/apps-metrics | AbilityRead | handleBatchAppMetrics |
| GET | /api/v1/node-providers | AbilityRead | handleListNodeProviders |
| POST | /api/v1/node-providers | AbilityWriteSensitive | handleSetNodeProviderCredential |
| GET | /api/v1/node-providers//regions | AbilityRead | handleListNodeProviderRegions |
| GET | /api/v1/node-providers//sizes | AbilityRead | handleListNodeProviderSizes |
| GET | /api/v1/node-provisions | AbilityRoot | handleListNodeProvisions |
| GET | /api/v1/node-provisions/ | AbilityRoot | handleGetNodeProvision |
| GET | /.well-known/jwks.json | Public | handleOIDCJWKS |
| GET | /api/v1/pipelines/oidc | AbilityRead | handleGetPipelineOIDCInfo |
| POST | /api/v1/pipelines/oidc/rotate-key | AbilityRoot | handleRotatePipelineOIDCKey |
| GET | /api/v1/network/topology | AbilityRead | handleGetNetworkTopology |
| GET | /api/v1/ssh-node-provisions | AbilityRoot | handleListSSHNodeProvisions |
| GET | /api/v1/ssh-node-provisions/ | AbilityRoot | handleGetSSHNodeProvision |
| GET | /api/v1/network-shares | AbilityRead | handleListNetworkShares |
| POST | /api/v1/network-shares | AbilityWriteSensitive | handleCreateNetworkShare |
| GET | /api/v1/network-shares/ | AbilityRead | handleGetNetworkShare |
| PUT | /api/v1/network-shares/ | AbilityWriteSensitive | handleUpdateNetworkShare |
| DELETE | /api/v1/network-shares/ | AbilityWriteSensitive | handleDeleteNetworkShare |
| POST | /api/v1/network-shares//test | AbilityRead | handleTestNetworkShare |
| GET | /api/v1/firewall-rules | AbilityRead | handleListFirewallRules |
| POST | /api/v1/firewall-rules | AbilityWriteSensitive | handleCreateFirewallRule |
| DELETE | /api/v1/firewall-rules/ | AbilityWriteSensitive | handleDeleteFirewallRule |
| GET | /api/v1/openapi.json | AbilityRead | handleOpenAPISpec |
| GET | /api/v1/changelog | AbilityRead | handleGetChangelog |
| GET | /api/v1/templates/custom | AbilityRead | handleListCustomTemplates |
| DELETE | /api/v1/templates/custom/ | AbilityWrite | handleDeleteCustomTemplate |
| GET | /api/v1/network/proxy | AbilityRead | handleGetNetworkProxy |
| GET | /api/v1/firewall/host | AbilityRead | handleHostFirewallStatus |
| POST | /api/v1/firewall/host/enable | AbilityRoot | handleEnableHostFirewall |
| POST | /api/v1/firewall/host/disable | AbilityRoot | handleDisableHostFirewall |
| POST | /api/v1/hooks/image-update// | Public | handleImageUpdateWebhook |
| GET | /api/v1/hooks/wake | Public | handleWakeHook |
| GET | /api/v1/imports/platform/databases | AbilityRead | handleListDatabaseDataCopies |
| GET | /api/v1/imports/platform/databases/ | AbilityRead | handleGetDatabaseDataCopy |
| POST | /api/v1/imports/platform/databases//copy | AbilityWriteSensitive | handleCopyDatabaseData |
| GET | /api/v1/migration/volumes | AbilityRead | handleVolumeGuide |
| GET | /api/v1/migration/cutover | AbilityRead | handleCutoverReport |
| GET | /api/v1/migration/cutover/verify | AbilityWriteSensitive | handleCutoverVerify |
| GET | /api/v1/ingress/connectivity | AbilityRead | handleIngressConnectivity |
| GET | /api/v1/firewall/exposure | AbilityRead | handleExposureReport |
| POST | /api/v1/firewall/exposure/preview | AbilityRoot | handleExposurePreview |
| PUT | /api/v1/firewall/exposure/restrictions// | AbilityRoot | handleExposureRestrict |
| DELETE | /api/v1/firewall/exposure/restrictions// | AbilityRoot | handleExposureUnrestrict |
| GET | /api/v1/attention/feed | AbilityRead | handleAttentionFeed |
| POST | /api/v1/attention/dismiss | Session | handleAttentionDismiss |
| GET | /api/v1/external-databases | AbilityRead | handleListExternalDatabases |
| POST | /api/v1/external-databases | AbilityRoot | handleCreateExternalDatabase |
| POST | /api/v1/external-databases/test | AbilityRoot | handleTestExternalDatabase |
| GET | /api/v1/external-databases/candidates | AbilityRoot | handleListExternalDatabaseCandidates |
| POST | /api/v1/external-databases/adopt | AbilityRoot | handleAdoptExternalDatabase |
| GET | /api/v1/external-databases/ | AbilityRead | handleGetExternalDatabase |
| PUT | /api/v1/external-databases/ | AbilityRoot | handleUpdateExternalDatabase |
| DELETE | /api/v1/external-databases/ | AbilityWrite | handleDeleteExternalDatabase |
| POST | /api/v1/external-databases//probe | AbilityWrite | handleProbeExternalDatabase |
| GET | /api/v1/external-databases//password | AbilityRoot | handleRevealExternalDatabasePassword |
| GET | /api/v1/usage/summary | AbilityRead | handleUsageSummary |
| GET | /api/v1/migration/hub/local-sources | AbilityRead | handleHubLocalSources |
| POST | /api/v1/migration/hub/sessions | AbilityWriteSensitive | handleCreateHubSession |
| GET | /api/v1/migration/hub/sessions | AbilityRead | handleListHubSessions |
| GET | /api/v1/migration/hub/sessions/ | AbilityRead | handleGetHubSession |
| DELETE | /api/v1/migration/hub/sessions/ | AbilityWriteSensitive | handleDeleteHubSession |
| PUT | /api/v1/migration/hub/sessions//selection | AbilityWriteSensitive | handlePutHubSelection |
| PUT | /api/v1/migration/hub/sessions//step | AbilityWriteSensitive | handlePutHubStep |
| POST | /api/v1/migration/hub/sessions//apply | AbilityWriteSensitive | handleApplyHubSession |
| GET | /api/v1/migration/hub/sessions//receipt | AbilityRead | handleHubReceipt |
| GET | /api/v1/migration/hub/sessions//items//connection | AbilityRead | handleHubConnection |
| POST | /api/v1/migration/hub/sessions//items//reveal | AbilityReadSensitive | handleHubReveal |
| POST | /api/v1/migration/apps/plan | AbilityWriteSensitive | handleAppImportPlan |
| POST | /api/v1/migration/apps/sessions | AbilityWriteSensitive | handleCreateAppImportSession |
| GET | /api/v1/migration/apps/sessions | AbilityRead | handleListAppImportSessions |
| GET | /api/v1/migration/apps/sessions/ | AbilityRead | handleGetAppImportSession |
| DELETE | /api/v1/migration/apps/sessions/ | AbilityWriteSensitive | handleDeleteAppImportSession |
| POST | /api/v1/migration/apps/sessions//connect | AbilityWriteSensitive | handleConnectAppImportSession |
| PUT | /api/v1/migration/apps/sessions//plan | AbilityWriteSensitive | handlePutAppImportPlan |
| POST | /api/v1/migration/apps/sessions//stage | AbilityWriteSensitive | handleStageAppImport |
| POST | /api/v1/migration/apps/sessions//verify | AbilityWriteSensitive | handleVerifyAppImport |
| POST | /api/v1/migration/apps/sessions//rollback | AbilityWriteSensitive | handleRollbackAppImport |
| GET | /api/v1/migration/apps/sessions//images | AbilityRead | handleAppImportImages |
| GET | /api/v1/migration/apps/sessions//images/status | AbilityRead | handleAppImportImagesStatus |
| POST | /api/v1/migration/apps/sessions//images/transfer | AbilityWriteSensitive | handleTransferAppImportImages |
| POST | /api/v1/migration/apps/sessions//images/cancel | AbilityWriteSensitive | handleCancelAppImportImages |
| GET | /api/v1/migration/apps/sessions//volumes | AbilityRead | handleAppImportVolumes |
| PUT | /api/v1/migration/apps/sessions//items//volumes/ | AbilityWriteSensitive | handleSetAppImportVolume |
| GET | /api/v1/migration/apps/sessions//cutover | AbilityRead | handleAppImportCutover |
| GET | /api/v1/migration/apps/sessions//cutover/verify | AbilityWriteSensitive | handleAppImportCutoverVerify |
| POST | /api/v1/migration/apps/sessions//items//route | AbilityWriteSensitive | handleRouteAppImport |
| GET | /api/v1/migration/apps/sessions//receipt | AbilityRead | handleAppImportReceipt |
| GET | /api/v1/dns/zones | AbilityRead | handleListDNSZones |
| POST | /api/v1/dns/zones | AbilityRoot | handleCreateDNSZone |
| GET | /api/v1/dns/zones/ | AbilityRead | handleGetDNSZone |
| DELETE | /api/v1/dns/zones/ | AbilityRoot | handleDeleteDNSZone |
| GET | /api/v1/dns/zones//nameservers | AbilityRead | handleDNSZoneNameServers |
| GET | /api/v1/dns/zones//delegation | AbilityRead | handleDNSZoneDelegation |
| GET | /api/v1/dns/zones//discover | AbilityRead | handleDiscoverDNSZoneRecords |
| GET | /api/v1/dns/zones//records | AbilityRead | handleListDNSZoneRecords |
| POST | /api/v1/dns/zones//records | AbilityRoot | handleCreateDNSZoneRecord |
| PUT | /api/v1/dns/zones//records | AbilityRoot | handleUpdateDNSZoneRecord |
| DELETE | /api/v1/dns/zones//records | AbilityRoot | handleDeleteDNSZoneRecord |
| POST | /api/v1/dns/zones//records/import | AbilityRoot | handleImportDNSZoneRecords |
| GET | /api/v1/dns/zones//records/export | AbilityRead | handleExportDNSZoneRecords |
| POST | /api/v1/dns/zones//templates/ | AbilityRoot | handleApplyDNSTemplate |
| GET | /api/v1/dns/templates | AbilityRead | handleListDNSTemplates |
| GET | /api/v1/dns/check | AbilityRead | handleDNSCheck |
| GET | /api/v1/dns/health-checks | AbilityRead | handleListDNSHealthChecks |
| POST | /api/v1/dns/health-checks | AbilityRoot | handleCreateDNSHealthCheck |
| DELETE | /api/v1/dns/health-checks/ | AbilityRoot | handleDeleteDNSHealthCheck |
| GET | /api/v1/dns/zone | AbilityRead | handleDNSZone |
| GET | /api/v1/traffic/summary | AbilityRead | handleTrafficSummary |
| GET | /api/v1/migration/apps/sessions//items//cutover/plan | AbilityRead | handleAppImportCutoverPlan |
| GET | /api/v1/migration/apps/sessions//items//cutover/runs | AbilityRead | handleListAppImportCutoverRuns |
| POST | /api/v1/migration/apps/sessions//items//cutover/runs | AbilityWriteSensitive | handleStartAppImportCutover |
| GET | /api/v1/migration/apps/sessions//items//cutover/runs/ | AbilityRead | handleGetAppImportCutoverRun |
| POST | /api/v1/migration/apps/sessions//items//cutover/runs//rollback | AbilityWriteSensitive | handleRollbackAppImportCutover |
| POST | /api/v1/migration/apps/sessions//items//cutover/runs//confirm-dns | AbilityWriteSensitive | handleConfirmAppImportCutoverDNS |
| GET | /api/v1/compose | AbilityRead | handleListComposeStacks |
| POST | /api/v1/compose/preview | AbilityDeploy | handlePreviewCompose |
| GET | /api/v1/node-moves | AbilityRoot | handleListNodeMoves |
| GET | /api/v1/node-moves/ | AbilityRoot | handleGetNodeMove |
| GET | /api/v1/node-drains/ | AbilityRoot | handleGetNodeDrain |
| GET | /api/v1/mesh/connectivity | AbilityRoot | handleMeshConnectivity |
| GET | /api/v1/teams | AbilityRead | handleListTeams |
| POST | /api/v1/teams | AbilityRoot | handleCreateTeam |
| GET | /api/v1/teams/roles | AbilityRead | handleListTeamRoles |
| GET | /api/v1/teams/ | AbilityRead | handleGetTeam |
| PATCH | /api/v1/teams/ | AbilityWrite | handleRenameTeam |
| DELETE | /api/v1/teams/ | AbilityWrite | handleDeleteTeam |
| GET | /api/v1/teams//members | AbilityRead | handleListTeamMembers |
| PUT | /api/v1/teams//members/ | AbilityWrite | handleSetTeamMemberRole |
| DELETE | /api/v1/teams//members/ | AbilityWrite | handleRemoveTeamMember |
| DELETE | /api/v1/teams//members/me | AbilityRead | handleLeaveTeam |
| POST | /api/v1/teams//transfer | AbilityWrite | handleTransferTeam |
| GET | /api/v1/teams//events | AbilityRead | handleListTeamEvents |
| GET | /api/v1/teams//invites | AbilityRead | handleListTeamInvites |
| POST | /api/v1/teams//invites | AbilityWrite | handleCreateTeamInvite |
| DELETE | /api/v1/teams//invites/ | AbilityWrite | handleRevokeTeamInvite |
| GET | /api/v1/team-invites/ | Session | handlePreviewTeamInvite |
| POST | /api/v1/team-invites/accept | Public | handleAcceptTeamInvite |
| GET | /metrics | AbilityRead | handleScrapeMetrics |
| GET | /api/v1/export/destinations | AbilityRoot | handleListExportDestinations |
| POST | /api/v1/export/destinations | AbilityRoot | handleCreateExportDestination |
| GET | /api/v1/export/destinations/ | AbilityRoot | handleGetExportDestination |
| PATCH | /api/v1/export/destinations/ | AbilityRoot | handleUpdateExportDestination |
| DELETE | /api/v1/export/destinations/ | AbilityRoot | handleDeleteExportDestination |
| POST | /api/v1/export/destinations//test | AbilityRoot | handleTestExportDestination |
| GET | /api/v1/mesh/settings | AbilityRead | handleGetMeshSettings |
| PUT | /api/v1/mesh/settings | AbilityRoot | handleUpdateMeshSettings |
| GET | /api/v1/resource-advisor/findings | AbilityRead | handleResourceAdvisorFindings |
| GET | /api/v1/resource-advisor/summary | AbilityRead | handleResourceAdvisorSummary |
| POST | /api/v1/resource-advisor/findings//apply | AbilityWrite | handleResourceAdvisorApply |
| POST | /api/v1/resource-advisor/findings//dismiss | AbilityWrite | handleResourceAdvisorState |
| POST | /api/v1/resource-advisor/findings//snooze | AbilityWrite | handleResourceAdvisorState |
| POST | /api/v1/resource-advisor/findings//restore | AbilityWrite | handleResourceAdvisorState |
| POST | /api/v1/imports/analyze | AbilityWrite | handleAnalyzeImport |
| GET | /api/v1/storage/orphans | AbilityRead | handleListStorageOrphans |
| DELETE | /api/v1/storage/orphans/ | AbilityWriteSensitive | handleReclaimStorageOrphan |
| GET | /api/v1/setup/guide | AbilityRead | handleGetSetupGuide |
| PUT | /api/v1/setup/guide/steps/ | AbilityRoot | handleSetSetupGuideStep |
| PUT | /api/v1/setup/guide/visibility | AbilityRoot | handleSetSetupGuideVisibility |
| POST | /api/v1/setup/guide/reset | AbilityRoot | handleResetSetupGuide |
| GET | /api/v1/logs/queries | AbilityRead | handleListSavedLogQueries |
| POST | /api/v1/logs/queries | AbilityWrite | handleCreateSavedLogQuery |
| PUT | /api/v1/logs/queries/ | AbilityWrite | handleUpdateSavedLogQuery |
| DELETE | /api/v1/logs/queries/ | AbilityWrite | handleDeleteSavedLogQuery |
| GET | /api/v1/usage/report | AbilityRead | handleUsageReport |
| GET | /api/v1/usage/settings | AbilityRoot | handleGetUsageSettings |
| PUT | /api/v1/usage/settings | AbilityRoot | handlePutUsageSettings |
| GET | /api/v1/usage/budgets | AbilityRead | handleListUsageBudgets |
| PUT | /api/v1/usage/budgets | AbilityWrite | handlePutUsageBudget |
| DELETE | /api/v1/usage/budgets/ | AbilityWrite | handleDeleteUsageBudget |
| POST | /api/v1/mobile/pairing-codes | AbilityRoot | handleCreateMobilePairing |
| POST | /api/v1/mobile/pair | Public | handleMobilePair |
| GET | /api/v1/mobile/devices | AbilityRead | handleListMobileDevices |
| DELETE | /api/v1/mobile/devices/ | AbilityRead | handleRevokeMobileDevice |
| GET | /api/v1/mobile/overview | AbilityRead | handleMobileOverview |
| GET | /api/v1/gate/check | Public | handleGateCheck |
| GET | /gate/login | Public | handleGateLogin |
| GET | /api/v1/demo | Public | handleDemoInfo |
| POST | /api/v1/demo/session | Public | handleDemoSession |
| POST | /api/v1/demo/cta | Public | handleDemoCTA |
| GET | /api/v1/deployments/timeline | AbilityRead | handleDeploymentsTimeline |
| GET | /api/v1/event-webhooks | AbilityRoot | handleListEventWebhooks |
| POST | /api/v1/event-webhooks | AbilityRoot | handleCreateEventWebhook |
| GET | /api/v1/event-webhook-types | AbilityRoot | handleListEventWebhookTypes |
| GET | /api/v1/event-webhooks/ | AbilityRoot | handleGetEventWebhook |
| PATCH | /api/v1/event-webhooks/ | AbilityRoot | handleUpdateEventWebhook |
| DELETE | /api/v1/event-webhooks/ | AbilityRoot | handleDeleteEventWebhook |
| POST | /api/v1/event-webhooks//test | AbilityRoot | handleTestEventWebhook |
| POST | /api/v1/event-webhooks//rotate-secret | AbilityRoot | handleRotateEventWebhookSecret |
| GET | /api/v1/event-webhooks//deliveries | AbilityRoot | handleListEventWebhookDeliveries |
| POST | /api/v1/event-webhook-deliveries//replay | AbilityRoot | handleReplayEventWebhookDelivery |
| GET | /api/v1/secret-rotation | AbilityRoot | handleListAllSecretRotation |
| GET | /api/v1/teams//quota | AbilityRead | handleGetQuota |
| PUT | /api/v1/teams//quota | AbilityWrite | handleSetQuota |
| DELETE | /api/v1/teams//quota | AbilityWrite | handleDeleteQuota |
| GET | /api/v1/scheduled-maintenance | AbilityRead | handleListScheduledMaintenance |
| POST | /api/v1/scheduled-maintenance | AbilityWriteSensitive | handleCreateScheduledMaintenance |
| DELETE | /api/v1/scheduled-maintenance/ | AbilityWriteSensitive | handleDeleteScheduledMaintenance |
| GET | /api/v1/compliance/access-reviews | AbilityRoot | handleListAccessReviews |
| POST | /api/v1/compliance/access-reviews | AbilityRoot | handleCreateAccessReview |
| GET | /api/v1/compliance/access-reviews/ | AbilityRoot | handleGetAccessReview |
| PUT | /api/v1/compliance/access-reviews//items/ | AbilityRoot | handleDecideAccessReviewItem |
| POST | /api/v1/compliance/access-reviews//complete | AbilityRoot | handleCompleteAccessReview |
| GET | /api/v1/compliance/evidence-packs | AbilityRoot | handleListEvidencePacks |
| POST | /api/v1/compliance/evidence-packs | AbilityRoot | handleCreateEvidencePack |
| GET | /api/v1/compliance/evidence-packs/ | AbilityRoot | handleGetEvidencePack |
| GET | /api/v1/compliance/evidence-packs//download | AbilityRoot | handleDownloadEvidencePack |
| POST | /api/v1/compliance/evidence-packs//verify | AbilityRoot | handleVerifyEvidencePack |
| GET | /api/v1/sso/saml/connections | AbilityRoot | handleListSAMLConnections |
| POST | /api/v1/sso/saml/connections | AbilityRoot | handleCreateSAMLConnection |
| GET | /api/v1/sso/saml/connections/ | AbilityRoot | handleGetSAMLConnection |
| PUT | /api/v1/sso/saml/connections/ | AbilityRoot | handleUpdateSAMLConnection |
| DELETE | /api/v1/sso/saml/connections/ | AbilityRoot | handleDeleteSAMLConnection |
| POST | /api/v1/sso/saml/connections//metadata | AbilityRoot | handleImportSAMLMetadata |
| GET | /api/v1/sso/group-mappings | AbilityRoot | handleListSSOGroupMappings |
| PUT | /api/v1/sso/group-mappings | AbilityRoot | handlePutSSOGroupMappings |
| GET | /api/v1/sso/settings | AbilityRoot | handleGetSSOSettings |
| PUT | /api/v1/sso/settings | AbilityRoot | handlePutSSOSettings |
| GET | /api/v1/sso/scim/tokens | AbilityRoot | handleListSCIMTokens |
| POST | /api/v1/sso/scim/tokens | AbilityRoot | handleCreateSCIMToken |
| POST | /api/v1/sso/scim/tokens//rotate | AbilityRoot | handleRotateSCIMToken |
| DELETE | /api/v1/sso/scim/tokens/ | AbilityRoot | handleRevokeSCIMToken |
| GET | /api/v1/scim/v2/ServiceProviderConfig | Public | handleSCIM |
| GET | /api/v1/scim/v2/ResourceTypes | Public | handleSCIM |
| GET | /api/v1/scim/v2/Users | Public | handleSCIM |
| POST | /api/v1/scim/v2/Users | Public | handleSCIM |
| GET | /api/v1/scim/v2/Users/ | Public | handleSCIM |
| PUT | /api/v1/scim/v2/Users/ | Public | handleSCIM |
| PATCH | /api/v1/scim/v2/Users/ | Public | handleSCIM |
| DELETE | /api/v1/scim/v2/Users/ | Public | handleSCIM |
| GET | /api/v1/scim/v2/Groups | Public | handleSCIM |
| POST | /api/v1/scim/v2/Groups | Public | handleSCIM |
| GET | /api/v1/scim/v2/Groups/ | Public | handleSCIM |
| PUT | /api/v1/scim/v2/Groups/ | Public | handleSCIM |
| PATCH | /api/v1/scim/v2/Groups/ | Public | handleSCIM |
| DELETE | /api/v1/scim/v2/Groups/ | Public | handleSCIM |
| POST | /api/v1/node-providers//test | AbilityWriteSensitive | handleTestNodeProvider |
| DELETE | /api/v1/node-providers/ | AbilityWriteSensitive | handleDisconnectNodeProvider |
| GET | /api/v1/registry-vendors | AbilityRead | handleListRegistryVendors |
| GET | /api/v1/package-registries | AbilityRead | handleListPackageRegistries |
| POST | /api/v1/package-registries | AbilityWriteSensitive | handleCreatePackageRegistry |
| DELETE | /api/v1/package-registries/ | AbilityWriteSensitive | handleDeletePackageRegistry |
| POST | /api/v1/package-registries//test | AbilityWriteSensitive | handleTestPackageRegistry |
| GET | /api/v1/database-versions | AbilityRead | handleListDatabaseVersions |
| POST | /api/v1/database-versions/refresh | AbilityWriteSensitive | handleRefreshDatabaseVersions |
| GET | /api/v1/database-catalog | AbilityRead | handleGetDatabaseCatalogStatus |
| POST | /api/v1/database-catalog/refresh | AbilityWriteSensitive | handleRefreshDatabaseCatalog |
| GET | /api/v1/scim/v2/ResourceTypes/ | Public | handleSCIM |
| GET | /api/v1/scim/v2/Schemas | Public | handleSCIM |
| GET | /api/v1/scim/v2/Schemas/ | Public | handleSCIM |
| GET | /api/v1/sleep | AbilityRead | handleSleepFleet |
| GET | /api/v1/deploy-policies | AbilityRead | handleListDeployPolicies |
| POST | /api/v1/deploy-policies/check | AbilityRead | handleCheckDeployPolicy |
| PUT | /api/v1/deploy-policies// | AbilityRoot | handleSetDeployPolicy |
| DELETE | /api/v1/deploy-policies// | AbilityRoot | handleDeleteDeployPolicy |
| POST | /api/v1/otlp/v1/traces | Public | handleOTLPTraces |
| POST | /api/v1/sentry/api//envelope/ | Public | handleSentryEnvelope |
| POST | /api/v1/sentry/api//store/ | Public | handleSentryStore |
| OPTIONS | /api/v1/sentry/api//envelope/ | Public | handleSentryPreflight |
| OPTIONS | /api/v1/sentry/api//store/ | Public | handleSentryPreflight |
| GET | /api/v1/logs/usage | AbilityRead | handleLogUsage |
| GET | /api/v1/sleep/overview | AbilityRead | handleSleepOverview |
| GET | /api/v1/fixes | AbilityRead | handleListFixes |
| POST | /api/v1/fixes//preview | AbilityRead | handlePreviewFix |
| POST | /api/v1/fixes//apply | AbilityRoot | handleApplyFix |
| GET | /api/v1/build-limits | AbilityRead | handleGetBuildLimits |
| PUT | /api/v1/build-limits | AbilityRoot | handleSetBuildLimits |
| DELETE | /api/v1/build-limits | AbilityRoot | handleResetBuildLimits |
| GET | /api/v1/maintenance/domains | AbilityRead | handleListFleetMaintenance |
| POST | /api/v1/maintenance/preview | AbilityRead | handlePreviewMaintenance |
| POST | /api/v1/git-sources/analyze | AbilityWrite | handleAnalyzeImport |
| POST | /api/v1/hooks/pipeline// | Public | handlePipelineWebhook |
| GET | /api/v1/ai/usage | AbilityRead | handleAIUsage |
| GET | /api/v1/model-catalog | AbilityRead | handleModelCatalog |
## See also
* [Feature Catalog](/feature-catalog) - high-level overview of platform capabilities
* [Observability](/observability) - metrics, logs, alerts, and telemetry APIs
* [Architecture](/architecture) - control plane design and reconciliation patterns
* [CLI Reference](/cli-reference) - command-line tool for all API operations
# App ports and exposure
Source: https://docs.levelrail.com/app-exposure
Change an app's container port live, choose between proxy-only and direct IP and port access, and test reachability, without port fights between apps.
Many apps declare the same container port (3000, 8080). That is fine: Docker gives each container its own host port, and the embedded ingress routes by hostname on ports 80 and 443, so apps never compete for those. The **Network** tab of an app adds the controls for the cases where you want more.
## Modes
| Mode | What it does | URLs |
| - | - | - |
| Through the proxy (default) | The app is only reachable through the ingress, by its domain or the automatic `sslip.io` hostname. Other apps on the server reach it by name. No host port is opened. | `https://app.example.com`, `https://web.1-2-3-4.sslip.io` |
| Direct IP and port | The container is also published on a host port on every interface, and an allow rule for that port is added to the [host firewall](/host-firewall). | `http://203.0.113.10:24100` |
Direct mode is plain HTTP. It bypasses TLS, basic auth and sign-in gates configured on the domain, so use it to check that an app runs, not as its public address. It is only available for apps on the control plane's own node.
## Change the port or mode
In the app's Network tab, edit the container port and pick a mode, then choose **Preview changes**. The preview lists exactly what changes (container port, host port, bind address, firewall rule) and whether the container is recreated. **Apply** writes the desired state and the reconcilers do the rest: the proxy is retargeted, and only the container is recreated (the image is not rebuilt or pulled). A running container whose published port or bind address no longer matches the desired state is removed and recreated on the next reconcile pass (a change of container port, host port or bind address always does this); `PUT .../exposure` answers with `outcome` of `unchanged`, `applied` or `recreating`. Pinned host ports cannot overlap, so a brief gap during the recreate is expected. Creating or updating an app with a `host_port` that another app, the control plane or a local process holds is refused with 409 naming the holder. While a redeploy is in flight, `GET .../network` keeps reporting the serving container's host port with `rollout_in_progress: true`. Every change is recorded in the app's activity. **Switch back to proxy only** reverses it in one click and removes the firewall rule it added.
```bash theme={"dark"}
levelrail-cli apps exposure web # show mode, URLs, host ports in use
levelrail-cli apps exposure web --port 8000 --preview # dry run
levelrail-cli apps exposure web --mode direct --host-port 24100 # publish and open the firewall
levelrail-cli apps exposure web --mode proxied # back to proxy only
levelrail-cli apps exposure web check # probe reachability
```
`apps ports` is an alias of `apps exposure`. The API is `GET|PUT /api/v1/apps/{name}/exposure`, plus `POST .../exposure/preview` and `POST .../exposure/check`.
## Host port choice and conflicts
Leave the host port empty and a free one is picked from `20000-29999` (override with `APP_EXPOSURE_PORT_RANGE=lo-hi`), below Docker's own ephemeral range. The pick avoids ports held by this control plane (80, 443, 8080, 8088, 8443, 9443, the configured ingress and SSH ports), pinned ports and streams of other apps, ports bound by other running containers, and anything the kernel reports as listening on the host. A port you choose that is held is refused with who holds it. The table at the bottom of the panel lists the ports in use by apps and the control plane.
## Test reachability
**Test from here** connects to the container's published port and requests the app's readiness path (or `/`), then does the same through the direct port and through the ingress URL. It reports the status code and latency, and which hop failed:
| Failure | Meaning |
| - | - |
| Container not running | Nothing to reach. Start or redeploy the app. |
| Container not listening | Up, but nothing answers on the published port: wrong container port, or still starting. |
| Firewall closed | The server address does not answer on the direct port: host firewall or cloud security group. |
| Certificate not issued | The ingress serves its temporary certificate while the public CA is retried. |
| Proxy route missing | The ingress answered 404: the domain is not attached to this app. |
| Upstream failing | The route exists but the app returns 502 to 504. |
The probe runs from the control plane host only. It cannot prove a port is open to the whole internet: use an outside check for that.
## Certificates when many apps start at once
Automatic hostnames and domains are issued by the embedded ACME client, which serializes challenges on the single port 80 and 443 listeners and backs off on its own after a failure. If the public CA rejects or throttles a host (a rate limit, closed ports, DNS), the host keeps answering over HTTPS with a certificate from the platform's own CA, and the public CA is retried on the fallback certificate's renewal cycle (`APP_INGRESS_FALLBACK_CERT_LIFETIME`, default `6h`). The reason (`rate_limited`, `unreachable`, `dns`, `caa`) is shown next to the certificate. See [Domains and ingress](/domains-and-ingress#firewall-ports-80-and-443).
# app.yaml reference
Source: https://docs.levelrail.com/app-spec-reference
Complete reference for app.yaml - the declarative app spec for Levelrail
`app.yaml` is the declarative spec Levelrail reads from your repo: services, their build and runtime settings, and the managed databases they use. The control plane validates every field below against the JSON Schema embedded in it (`internal/spec/schema/app.schema.json`). An unknown key is an error, not something that gets quietly ignored.
Levelrail looks for the file under these names, in this order: `app.yaml`, `app.yml`, `deploy.yaml`, `deploy.yml`, then the product's own name with a `.yaml` or `.yml` extension. The generic names come first so a repo that already committed `app.yaml` keeps working.
## Full example
```yaml theme={"dark"}
version: 1
services:
web:
build:
type: dockerfile # dockerfile | compose | railpack | static | image
path: ./Dockerfile
baseDirectory: apps/web # build context subdirectory, for monorepos
domains:
- app.example.com
port: 3000
host_port: 30001 # pin the host-side port; omit to let Docker assign one
bind_address: private # private (default, loopback only) | public | a literal IP
health:
readiness: { path: /healthz, interval: 5s, timeout: 2s }
liveness: { path: /healthz, interval: 30s, failures: 3 }
readyTimeout: 90s # how long a fresh deploy waits for readiness before failing; default 60s
resources:
memory: 512Mi
cpu: 0.5
env:
DATABASE_URL: { from: postgres.main.url }
API_KEY: { secret: true, required: true }
STRIPE_KEY: { vault: { path: myapp/config, key: stripe_key } }
LOG_LEVEL: debug # plain string shorthand for a literal value
replicas: 2
strategy: rolling # rolling | recreate | blue-green
labels:
team: platform
tier: frontend
volumes:
- name: data # named Docker volume
path: /var/lib/data
- hostPath: /srv/web/uploads # bind mount of a real host directory
path: /uploads
readOnly: true
hooks:
preDeploy: rails db:migrate
postDeploy: curl -f https://hooks.example.com/deployed
command: ["node", "server.js", "--port", "3000"] # overrides the image's own CMD
egress:
mode: allowlist # outbound traffic limited to the hosts below
allow:
- { host: api.stripe.com, port: 443 }
dependsOn: [worker] # start order across sibling services
databases:
main:
engine: postgres # postgres | redis | mysql | mongodb | mariadb | keydb | clickhouse | dragonfly | valkey
version: "16"
backup: { schedule: "0 3 * * *", retain: 7 }
ephemeralInPreviews: true # opt in to a disposable per-pull-request instance, see below
```
A prebuilt image needs no build block: `image: ghcr.io/your-org/web:1.0` on the service is shorthand for `build: { type: image, image: ... }`.
## Field reference
### Top level (`Spec`)
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `version` | integer | yes | none | Must be exactly `1` (`schema: const: 1`). |
| `services` | map of name to `Service` | yes | none | At least one service required. Keys must match `^[a-z][a-z0-9-]*$` (lowercase alphanumeric and hyphens, starting with a letter). |
| `databases` | map of name to `Database` | no | none | Same key pattern as `services`. |
### `Service` (an entry under `services`)
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `build` | `Build` | one of `build`/`image` | none | How the service's image gets built. |
| `image` | string | one of `build`/`image` | none | Shorthand for `build: {type: image, image: ...}`: deploy a prebuilt image as-is. Works with `replicas`, `strategy` and every other service field. Setting both is an error unless `build.type` is `image` with the same reference. |
| `domains` | list of string | no | none | Public hostnames routed to this service. A domain can only be claimed by one service across the whole spec. |
| `port` | integer | conditional | none | 1 to 65535. Required unless `build.type` is `static`; must be omitted when `build.type` is `static`, since a static site has no running container to route to. |
| `host_port` | integer | no | auto-assigned | 1 to 65535. Pins the host-side port Docker binds `port` to. Omit to let Docker assign one. Must be omitted when `build.type` is `static`. |
| `bind_address` | string | no | `private` | `private` (loopback only), `public` (every interface, an explicit opt-in), or a literal IP (a specific host interface or a WireGuard mesh peer address). Picks which network interface `port` (and `host_port`, if pinned) publishes to; see [Bind addresses and exposure](#bind-addresses-and-exposure) below. Must be omitted when `build.type` is `static`. |
| `health` | `Health` | no | none | Readiness and liveness probe configuration. |
| `resources` | `Resources` | no | none | Memory and CPU limits. |
| `env` | map of name to `EnvVar` | no | none | Environment variables. See the `EnvVar` shapes below. |
| `replicas` | integer | no | `1` | Minimum 1 if set. |
| `strategy` | string | no | `blue-green` | One of `rolling`, `recreate`, `blue-green`. |
| `labels` | map of string to string | no | none | Arbitrary operator-supplied Docker labels applied to the container at create time. See Validation below for the limits enforced on these. |
| `volumes` | list of `Volume` | no | none | Named Docker volumes and host-directory bind mounts this service's container mounts. |
| `hooks` | `Hooks` | no | none | Pre/post-deploy commands run inside the container. Not meaningful when `build.type` is `static` or `compose`. |
| `jobs` | list of `Job` | no | none | Cron-scheduled commands, each run in a fresh container of this service's image or exec'd into it. See [Scheduled jobs](/scheduled-jobs). Not meaningful when `build.type` is `static` or `compose`. |
| `command` | list of string | no | none | Overrides the image's own default `CMD`. A plain argv list, never shell-interpreted. |
| `egress` | `Egress` | no | open egress | Limits outbound traffic to an allowlist of host and port pairs. See [`Egress`](#egress). |
| `loadbalancer` | `LoadBalancer` | no | none | Balances traffic across the service's replicas. See [Load balancing](/load-balancing). |
| `dependsOn` | list of string | no | none | Sibling `services:` keys this service waits on: the reconciler does not create this service's container until every named dependency has at least one running container. Start order only, real Docker Compose's own default `depends_on:` semantic (`service_started`), not a wait for the dependency's own readiness or health check. Each entry must name a real sibling service with a single running container (not `static` or `compose`), and a cycle between services fails validation. |
### `Build`
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `type` | string | yes | none | One of `dockerfile`, `compose`, `railpack`, `static`, `image`. |
| `path` | string | conditional | none | Required when `type` is `compose`. Optional for `dockerfile`, `railpack` and `static` (for example, a non-default Dockerfile path). Rejected for `image`. |
| (static deploys) | | | | A `static` deploy has no container, so it skips the readiness gate: it fetches the source, copies it to the embedded file server, records a content digest (which makes `rollback-to` work), logs the fetch and publish steps in the build log, and settles as soon as the site is published. |
| `baseDirectory` | string | no | repo root | Subdirectory used as the build context, for a monorepo (`apps/web`). Must be relative and stay inside the repo. Rejected for `image` and `compose` (a compose file's own `context:` already scopes each service). |
| `image` | string | conditional | none | Full registry reference such as `ghcr.io/org/app:v1.2.3`. Required when `type` is `image`, rejected otherwise. Nothing is built; the image is deployed as is. |
| `registryCredential` | string | no | none | Name of a registry credential saved in Levelrail, used to pull `image` from a private registry. Empty means an unauthenticated pull. |
| `args` | map of string to string | no | none | Dockerfile build-time `ARG` values, passed through to BuildKit as `--build-arg` equivalents. Only meaningful when `type` is `dockerfile`. |
### `Egress`
Opt-in outbound allowlist for one service. Without it the service's outbound traffic is unrestricted. Not valid with `build.type` of `static` or `compose`.
| Field | Type | Required | Description |
| - | - | - | - |
| `mode` | string | yes | Only `allowlist` is supported. |
| `allow` | list of `{host, port}` | yes, at least one entry | `host` is a hostname or IPv4 address (re-resolved periodically, not pinned at deploy time). `port` is 1 to 65535. |
### `Volume` (an entry under `volumes`)
Exactly one of `name` or `hostPath` must be set:
* `name`: declares a named Docker volume (scoped to this service)
* `hostPath`: a bind mount of a real directory on the node where the service runs
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `name` | string | conditional | none | A named Docker volume, scoped to this service (two services can each declare a volume named `data` without colliding). Must match `^[a-z][a-z0-9-]*$`. |
| `hostPath` | string | conditional | none | An absolute path on the host to bind-mount. Rejected if relative or under a protected system path (see Validation below). |
| `path` | string | yes | none | The container-side mount path. |
| `readOnly` | boolean | no | `false` | Mounts read-only inside the container. Only meaningful with `hostPath`; rejected on a named volume. |
**Constraint**: Two volumes on the same service can never share a `path`.
### `Health`
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `readiness` | `Probe` | no | none | Checked before cutting traffic to a new container. See [Health checks](#health-checks). |
| `liveness` | `Probe` | no | none | Checked on a running container to detect a crashloop. |
| `readyTimeout` | string | no | `60s` | Duration string matching `^[0-9]+(ms\|s\|m\|h)$`. How long a fresh deploy waits for `readiness` to pass before the deploy is marked `ReadinessFailed`. Raise this for a service with a genuinely slow cold start (JVM warm-up, a large migration, a slow external connection) instead of it being falsely flagged as failed; the platform still self-heals on the next resync once the container is actually healthy, but this avoids the false signal in the meantime. |
### `Probe`
A probe is either an HTTP(S) request (set `path`) or a command run inside the container (set `exec`), never both.
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `path` | string | one of `path`/`exec` | none | HTTP path to request, starting with `/`. |
| `scheme` | string | no | `http` | `http` or `https`. |
| `host` | string | no | none | `Host` header to send, and the TLS server name for certificate verification. A bare hostname or `host:port`. |
| `tls_skip_verify` | boolean | no | `false` | Accept any certificate. Only valid with `scheme: https`; meant for the self-signed certificates most images generate for themselves. Verification stays on unless this is set. |
| `follow_redirects` | boolean | no | follow | `false` judges a 3xx response itself against `expected_status` instead of following it. Left unset, redirects are followed (up to `APP_PROBE_MAX_REDIRECTS`, default 10), which is how probes behaved before this field existed. |
| `expected_status` | string, integer, or list | no | `200-299` | Accepted status codes: a code (`204`), a range (`200-399`), a comma-separated string (`"200,301-302"`), or a list (`[200, "301-302"]`). |
| `exec` | string or list | one of `path`/`exec` | none | A string runs through `/bin/sh -c`; a list runs as argv (for images with no shell). Exit code 0 means healthy. Runs through the Docker Engine API, bounded by `timeout`; its output (capped at `APP_PROBE_EXEC_OUTPUT_BYTES`, default 512) shows up in the failure reason. A command that hangs past `timeout` is reported as failed, but Docker has no API to kill an exec'd process, so it keeps running inside the container. |
| `interval` | string | no | `2s` (`APP_PROBE_DEFAULT_INTERVAL`) | Duration string matching `^[0-9]+(ms\|s\|m\|h)$`, for example `5s`. |
| `timeout` | string | no | `2s` (`APP_PROBE_DEFAULT_TIMEOUT`) | Same duration pattern as `interval`. |
| `failures` | integer | no | none | Minimum 1 if set. |
### Health checks
Readiness gates a deploy's cutover: the new container has to pass it before the old one is retired. Liveness runs on every reconcile pass against a running container and restarts it after `failures` consecutive failures. An HTTP probe needs a `port`; an exec probe does not, so a worker with nothing listening can still have one.
```yaml theme={"dark"}
health:
readiness:
path: /api/health
scheme: https
tls_skip_verify: true # self-signed certificate inside the container
follow_redirects: false
expected_status: 200-399 # a login redirect counts as up
liveness:
exec: pg_isready -U app # or [pg_isready, -U, app]
timeout: 5s
failures: 3
```
Every probe result lands in the app's `Ready` condition (reason `ReadinessFailed`, `RunningNotReady`, `LivenessDegraded`, and so on), and the message says exactly what failed, for example:
* `GET https://127.0.0.1:32768/health returned 302 to /login; set follow_redirects or expected_status (currently 200-299)`
* `GET https://127.0.0.1:32768/ : TLS verification failed: x509: certificate signed by unknown authority; set tls_skip_verify for a self-signed certificate`
* `exec "redis-cli -p 6390 ping" exited 1: Could not connect to Redis at 127.0.0.1:6390: Connection refused`
* `GET http://127.0.0.1:32768/healthz timed out after 2s`
The same settings are editable in the dashboard (app, then Health), with `levelrail-cli apps health set`, and through `PUT /api/v1/apps/{name}/health`.
A Compose file's `healthcheck:` is translated into a readiness probe: a `curl`/`wget` command becomes an HTTP(S) probe that keeps the tool's own semantics (`curl -L` follows redirects, `curl -f` accepts any status below 400, `-k`/`--no-check-certificate` skip TLS verification), `start_period` widens the readiness budget, and any other command (`pg_isready`, `redis-cli ping`, `mysqladmin ping`) becomes an exec probe. A bare `/dev/tcp` connect check is not translated.
### `Resources`
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `memory` | string | no | none | Pattern `^[0-9]+(Mi\|Gi)$`, for example `512Mi`. |
| `cpu` | number | no | none | Must be greater than 0 if set. |
| `swapMemory` | string | no | none | Same pattern as `memory`. Docker's `MemorySwap`: the combined memory+swap ceiling, not swap on top of memory, so it must be at least `memory` and requires `memory` to also be set. |
| `cpuSet` | string | no | none | Docker's `cpuset-cpus` format, for example `0-3` or `0,2`. Pins the container to specific host CPUs. |
| `gpu` | `all`, integer, or object | no | none | Requests NVIDIA GPUs. `all`, a count such as `2`, or `{count: 2}` / `{devices: ["0", "GPU-uuid"]}` (device indexes or UUIDs win over count). The app is refused on a node without a GPU and the nvidia container runtime, and moving it to such a node is rejected. See [AI models](/ai-models#gpu-nodes). |
### `LoadBalancer`
All fields are optional. Durations are Go duration strings such as `500ms`, `5s` or `2m`. See [Load balancing](/load-balancing) for behavior.
| Field | Type | Description |
| - | - | - |
| `algorithm` | string | `round_robin` (default), `least_conn`, `ip_hash`, `uri_hash`, `cookie` or `weighted`. |
| `cookie_name` | string | Sticky cookie name. Only with `algorithm: cookie`, default `lb`. |
| `weights` | list of integer | 1 to 100 per replica, indexed by replica number. Only with `algorithm: weighted`. |
| `active_health` | object | `path` (required, starts with `/`), `interval`, `timeout` (shorter than `interval`), `passes`, `fails`, `expect_status`. |
| `passive_health` | object | `max_fails`, `fail_duration`. |
| `retries` | object | `count` (0 to 10), `try_duration`, `try_interval`. |
| `slow_start` | duration | Weight ramp for a new replica. Weighted only. |
| `drain_timeout` | duration | How long open streams survive a cutover. |
| `request_timeout` | duration | Upstream response header timeout. |
| `rate_limit` | object | `rps` (required), `burst`. Per client address. |
| `upstream_tls` | object | `server_name`, `insecure_skip_verify`. Speak HTTPS to replicas. |
### `Hooks`
Shell commands the reconciler runs inside the service's own container via the Docker Engine API's exec facility (`sh -c `), not by shelling out.
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `preDeploy` | string | no | none | Runs once per deploy, before the new container's readiness probe and before any old container is removed. A nonzero exit blocks the deploy: the new container is rolled back and whatever was previously running keeps serving. |
| `postDeploy` | string | no | none | Runs once per deploy, after the full replica set has cut over and any stale container has been removed. A nonzero exit is reported (`PostDeployHookFailed` reconcile condition) but never rolls back an already-successful cutover. |
**Rules**
* At least one of `preDeploy`/`postDeploy` must be set; an empty `hooks: {}` block is rejected.
* With more than one replica, each hook runs exactly once per deploy (against the first replica), not once per replica. Running a database migration once per replica would be wrong, even though every replica gets a fresh container from the same image.
* The most recent outcome of each hook (exit code, captured output) is queryable via `GET /api/v1/apps/{name}/hook-runs`.
### `Job` (an entry under `jobs`)
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `name` | string | yes | none | Unique within the service: lowercase letters, digits, `-` and `_`. |
| `schedule` | string | yes | none | 5-field cron expression. |
| `command` | list of string | yes | none | Argv list, never shell-interpreted. |
| `timezone` | string | no | `UTC` | IANA zone the schedule is read in. |
| `mode` | string | no | `fresh` | `fresh` (new container from the service's spec) or `exec` (inside the running container). |
| `concurrency` | string | no | `forbid` | `allow`, `forbid` or `replace`. |
| `timeout` | duration | no | 10m | Go duration such as `30s`, `10m`. |
| `retries` | integer | no | `0` | Retries after a failed attempt. |
| `retryBackoff` | duration | no | 10s | First retry delay, doubled each retry. |
| `catchUp` | string | no | `skip` | `skip` or `run-once`. |
| `catchUpLookback` | duration | no | 1h | How far back `run-once` still applies. |
| `cpu` | number | no | the service's | CPU cores for a fresh container. |
| `memory` | string | no | the service's | `256Mi` or `1Gi` form, fresh container only. |
| `warnAfter` | duration | no | none | Raises the running-too-long alert. |
| `enabled` | boolean | no | `true` | Keep the job but do not run it. |
See [Scheduled jobs](/scheduled-jobs) for how runs, retries and catch-up behave.
### `EnvVar` (an entry under `env`)
Each value under `env` is either a plain string scalar (a literal value) or
an object with these fields.
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `from` | string | no | none | References another resource's computed value, for example `postgres.main.url`. |
| `secret` | boolean | no | `false` | The operator provides this value at deploy time through envelope-encrypted secret storage; it is never written to `app.yaml` or the git repo. |
| `required` | boolean | no | `false` | Only meaningful alongside `secret: true`: fail the deploy if no value has been provided, rather than starting the container with the variable unset. |
| `vault` | `VaultRef` | no | none | Resolves this value live from an external HashiCorp Vault instance instead of Levelrail's own envelope-encrypted storage. Mutually exclusive with `from` and `secret`. See [external secrets: HashiCorp Vault](/deploying-apps#external-secrets-from-hashicorp-vault). |
The object form must set at least one of `from`, `secret`, or `vault`.
`vault` is mutually exclusive with both `from` and `secret`: a given env
var resolves its value from exactly one source.
### `VaultRef` (external secrets integration)
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `path` | string | yes | none | The secret's path in Vault's KV v2 engine, for example `myapp/config`. Resolved against the mount path configured under Settings > Vault (default `secret`). |
| `key` | string | yes | none | The field name inside that secret's data, for example `api_key`. |
```yaml theme={"dark"}
env:
API_KEY: { vault: { path: myapp/config, key: api_key } }
```
The value is read fresh from Vault immediately before the container is created and is never persisted by Levelrail.
If Vault is unreachable, not configured, or the secret/field doesn't exist, the deploy fails loudly rather than starting the container with the variable empty.
### `Database` (an entry under `databases`)
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `engine` | string | yes | none | One of `postgres`, `redis`, `mysql`, `mongodb`, `mariadb`, `keydb`, `clickhouse`, `dragonfly`, `valkey`. |
| `version` | string | no | none | For example `"16"`. |
| `backup` | `Backup` | no | none | Backup schedule. |
| `ephemeralInPreviews` | boolean | no | `false` | Provision a full, disposable database of its own for every pull-request preview, destroyed with the preview and with no restore path. Only meaningful once the app's git source has preview environments enabled; see [Git integrations](/git-integrations) and [Deploy previews](/deploy-previews). |
| `isolatedInPreviews` | boolean | no | `false` | Give each preview its own credential on the existing database (a Postgres role or a Redis ACL user scoped to its own key prefix) instead of a new instance. Postgres and Redis only. Ignored when `ephemeralInPreviews` is also set. |
### `Backup`
| Field | Type | Required | Default | Description |
| - | - | - | - | - |
| `schedule` | string | yes | none | A cron expression, for example `"0 3 * * *"`. |
| `retain` | integer | no | none | Minimum 1 if set. |
## Bind addresses and exposure
Every published port (a service's `port`/`host_port`, and a managed database's public-access port; see [managing databases](/managing-databases)) binds to `bind_address`, resolved by `internal/bindaddr.Resolve`:
| Value | Resolves to | Meaning |
| - | - | - |
| `private` (default, or omitted) | `127.0.0.1` | Reachable only from this host. On a remote node with a healthy WireGuard mesh, the main port binds to that node's mesh IP instead (never public), so the control plane's ingress can reach it; see [multi-node](/multi-node#routing-to-apps-on-remote-nodes). |
| `public` | `0.0.0.0` | Reachable from any network that can route to this host. Requires the literal string `public`; blank or malformed values never resolve here. |
| any other value | itself | Treated as a literal IP (a specific host interface or, once the WireGuard mesh lands, a mesh peer address). Must parse as a valid IP. |
### Default behavior
`private` is the default for anything created after this field shipped. A service or database that was already publicly exposed before the field existed was backfilled to `public` and keeps that exposure. The `private` default only applies to a newly created resource when a redeploy or public-access change leaves `bind_address` unset.
## Validation
`spec.Parse` (`internal/spec/spec.go`) runs two layers. Both must pass before a caller sees a `Spec`.
### Layer 1: Structural shape
Checked against the embedded JSON Schema at `internal/spec/schema/app.schema.json` (compiled and cached by `compiledAppSchema` in `internal/spec/schema.go`).
Coverage:
* Required fields
* Enums
* String patterns (durations, memory sizes)
* Numeric ranges
* Rejection of unknown keys
YAML is decoded generically and round-tripped through JSON so the schema validator sees the same shapes as real JSON input.
### Layer 2: Semantic rules
Checked by `(*Spec).Validate` in `internal/spec/validate.go` for anything the schema can't express because it depends on multiple fields.
**Names**
* Service and database names must match `^[a-z][a-z0-9-]*$`.
* A domain can only be claimed by one service in the whole spec.
**Build configuration**
* `build.path` is required when `build.type` is `compose`.
* `build.image` is required when `build.type` is `image` and rejected for any other type. `build.path` and `build.baseDirectory` are rejected for `image`, and `build.baseDirectory` is rejected for `compose`.
* `build.baseDirectory` must be relative and must not escape the repository root.
* `build.args` is only meaningful when `build.type` is `dockerfile`. Set with any other build type, it is rejected rather than silently ignored.
**Port configuration**
* `port` is required unless `build.type` is `static`.
* `port` is forbidden when `build.type` is `static`.
**Strategy**
* `strategy`, if set, must be one of the three known values (a redundant check behind the schema's enum; `Validate` is documented as safe to call on a hand-built `Spec` that never went through schema validation).
**Service labels**
Checked by `ValidateLabels` (`internal/spec/labels.go`):
* At most 32 labels per service
* Keys up to 255 characters
* Values up to 4096 characters
* No empty keys
* No key starting with the reserved prefix `platform-reserved.` (kept open for the platform's own bookkeeping labels)
**Resource limits**
* `resources.swapMemory` requires `resources.memory` to also be set.
* `resources.swapMemory` must be at least `resources.memory` (checked during deploy translation in `internal/deploy`, since both values need byte conversion). Docker's `MemorySwap` is the combined memory+swap ceiling, not swap on top of memory.
**Egress**
* `egress` is rejected when `build.type` is `static` or `compose`. `mode` must be `allowlist` with at least one `allow` entry, each a hostname or IPv4 address with a valid port.
**Dependencies**
* Each `dependsOn` entry must name another service in the file, not itself, and not a `static` or `compose` service. A cycle is rejected.
**Hooks**
* `jobs` is rejected when `build.type` is `static` or `compose`, for the same reason as `hooks`. Job names must be unique within the service, the schedule must parse as a cron expression, `timezone` must be a known IANA zone, and durations must be positive.
* `hooks` is rejected when `build.type` is `static` (no container to run a command in) or `compose` (a wrapper that expands into N real services; one `hooks` block cannot unambiguously target any of them).
* An empty `hooks: {}` block is rejected at the schema layer (the schema's own `minProperties: 1`).
**Volumes**
* Each volume's `name` (if a named volume) must match `^[a-z][a-z0-9-]*$`.
* No two volumes on the same service may share a name or the same `path`.
* Each volume's `hostPath` (if a bind mount) must be an absolute path.
* `hostPath` is rejected under protected system paths: `/`, `/etc`, `/root`, `/boot`, `/sys`, `/proc`, `/var/lib/docker`, `/var/run/docker.sock`, or `/var/run`. `/var/run/docker.sock` is excluded by design (it's a full container-escape-to-host-root vector; this feature doesn't grant that capability).
* `readOnly: true` on a volume with `name` set (not `hostPath`) is rejected (no meaning for a named Docker volume).
### Additional guard: strict YAML parsing
`spec.Parse` also runs `yamlUnmarshalStrict` (YAML decode with `KnownFields(true)`) as an independent guard against struct tags and JSON Schema drifting apart. An unknown key becomes a decode error during development rather than a silently dropped field in production.
## Next steps
Deploy your first app with an `app.yaml`.
Secrets, environment setup, rollouts and rollbacks.
Database configuration and backups.
Branch, push versus release trigger, and webhooks. None of that lives in `app.yaml`.
# Architecture
Source: https://docs.levelrail.com/architecture
Core architecture of Levelrail's reconciler, agent, builds, ingress, state, observability, and security layers.
This page explains how Levelrail is built, for contributors and for operators who want to know what runs on their server. It describes the code as it is today. To use the platform, start with [Getting started](/getting-started). For maturity per feature, see [feature status](/feature-status).
## Control plane
Package: `internal/reconcile` (see `internal/reconcile/engine.go`, and the per-resource controllers under `internal/reconcile/application`, `internal/reconcile/database`, `internal/reconcile/ingress`, `internal/reconcile/mesh`, `internal/reconcile/nodehealth`).
The core loop is a reconciler, not an imperative deploy script. Desired state lives in the database as declarative resource records. Each controller compares desired state against what Docker reports and makes the two match. The pattern comes from Kubernetes controllers, without the rest of the Kubernetes runtime.
Every controller is idempotent and level-triggered, never edge-triggered. `Reconcile` never assumes it's continuing a previous partial operation. On every call it works out what to do from current observed state alone.
That makes it safe to interrupt and safe to retry. Call it again after a half-finished attempt and it converges, instead of double-applying or corrupting state.
Every reconcile also emits a status condition with a reason string. A failure shows up in the UI with an explanation, not a spinner that never resolves.
```mermaid theme={"dark"}
flowchart TD
A["Desired state
(database records)"] -->|Read| B["Reconcile loop"]
C["Observed state
(Docker events)"] -->|Stream| B
B -->|Diff| D{Match?}
D -->|No| E["Converge
(issue commands)"]
D -->|Yes| F["No-op"]
E -->|Emit| G["Status condition
(reason + outcome)"]
F -->|Emit| G
G -->|Show in UI| H["Dashboard"]
E -.->|Watch| C
```
## Node agent
Packages: `internal/agent` (see `internal/agent/transport.go`, `internal/agent/grpc_transport.go`, `internal/agent/server.go`) and `internal/docker` (see `internal/docker/client.go`).
The node agent dials out to the control plane, not the other way around, so managed servers never need an inbound port open. It talks to the local Docker Engine API directly and never shells out to the `docker` CLI.
The transport is an interface (`agent.Transport`) with two implementations:
* `Local` - calls a `docker.Runtime` in-process (single-node)
* `GRPCTransport` - dispatches the same calls over a real reverse-dialed gRPC connection with mTLS ([multi-node](/multi-node))
Both ship today, along with node enrollment (one-time join tokens exchanged for client certificates), a real `levelrail-agent` binary, and cross-node service placement. Reconcilers and everything above the transport boundary don't know which implementation they're talking to, so single-node and multi-node run the same code path.
```mermaid theme={"dark"}
graph LR
R["Reconciler
(reconcile/*)"]
T["Transport interface
(agent.Transport)"]
L["Local
(in-process)"]
G["gRPC
(reverse-dialed)"]
D["Docker Engine API"]
A["Node agent"]
R -->|calls| T
T -->|impl| L
T -->|impl| G
L -->|talks to| D
G -->|reverse dial| A
A -->|talks to| D
```
Two more pieces sit alongside the agent:
* WireGuard mesh: `internal/network` (built on `wireguard-go`) is designed to give every node a peer and internal DNS names that resolve across machines. It is beta and not yet verified across real hosts, see [feature status](/feature-status)
* Dedicated build nodes: nodes marked build-capable run builds so they don't compete with production containers for CPU
## Builds
Package: `internal/build` (see `internal/build/client.go`, `internal/build/solve.go`).
Builds go through BuildKit's Go client, not `docker build`. That gives you a remote build cache and parallel stage execution, which the CLI path doesn't offer.
Three build types are supported:
* Dockerfile builds go through the `dockerfile.v0` frontend
* Railpack (`github.com/railwayapp/railpack`) detects a build plan for apps that don't ship a Dockerfile
* Static sites (`build.type: static`) take their own path with no container involved
The remote cache is registry-backed (`internal/build/cache.go`), with an S3 option (`internal/build/cache_s3.go`). Dedicated [build nodes](/build-node-routing) share it instead of each rebuilding from scratch.
## Ingress
Package: `internal/ingress` (see `internal/ingress/driver.go`, `internal/ingress/certstorage.go`).
Caddy (`github.com/caddyserver/caddy/v2`) is embedded as a library and driven in-process through its admin API, instead of running as a separate container with its own config. You get automatic TLS and HTTP/3 without a second moving part to operate.
Certificates and ACME account keys live in the database (`internal/ingress/certstorage.go` implements `certmagic.Storage` over `internal/store`), so certs are shared across nodes rather than pinned to whichever machine issued them. On first start after an upgrade, `internal/ingress/certimport.go` copies any certificates Caddy had written to its on-disk storage into the database, so nothing re-issues. Saving a different CA drops the other issuers' certificates (`internal/ingress/certpurge.go`), because certmagic reuses a valid certificate from any issuer.
`internal/ingress/publicip.go` resolves the server's public IP at startup (`APP_PUBLIC_HOST`, else a probe of public what-is-my-IP services). That gives zero-DNS hostnames: `.sslip.io` is built from it for the dashboard (one-click Enable HTTPS, `internal/api/ingress_https.go`) and `..sslip.io` for every app with no domain (`internal/ingress/fallback_domain.go`, toggle in `ingress_settings.fallback_domains_disabled`).
When the HTTP ingress port can be bound, Caddy's automatic redirect server owns it (`308` to https) and also serves HTTP-01 challenges. Otherwise the port stays closed outside challenges.
An `events` subscription (`internal/ingress/acme_events.go`) records certmagic's `cert_failed` events per hostname, so the API can report the CA's own error and a hint (`GET /api/v1/settings/ingress/https`).
TLS issuance works like this:
* Defaults to an internal, self-signed issuer
* A real Caddy ACME issuer, settings toggle, and form validation are built and wired end to end
* Verified: real Let's Encrypt issuance over `.sslip.io` on a public VPS (see [the recorded run](/acme-verification-runbook#recorded-run-2026-10-05)). Renewal and DNS-01 wildcards are still not exercised live
## State
Package: `internal/store` (see `internal/store/store.go`,
`internal/store/database.go`, `internal/store/migrate.go`).
State is embedded SQLite in WAL mode, opened via `modernc.org/sqlite`
(pure Go, no cgo), which keeps cross-compiling the control plane binary
simple. Schema changes go through embedded, versioned, forward-only
migrations.
## Observability
Package: `internal/telemetry` (see `internal/telemetry/collector.go`, `internal/telemetry/logs.go`, `internal/telemetry/federate.go`).
Each node keeps its own data:
* A metrics store: a local time-series store at 15s resolution (CPU, memory, disk IO, network IO, deploy count, build duration)
* A log store: local logs with full-text search (FTS5) and structured JSON log parsing
Nothing is centrally indexed by default, so idle cost stays near zero as you add apps.
Federated query is shipped, not planned. `internal/telemetry/federate.go` defines a `MetricsSource` interface that both the local, single-node store and a remote node's data satisfy identically. The query API (`internal/api`) fans a query out and merges results across nodes.
```mermaid theme={"dark"}
graph TD
N1["Node 1
Metrics + Logs
FTS5 index"]
N2["Node 2
Metrics + Logs
FTS5 index"]
N3["Node N
Metrics + Logs
FTS5 index"]
Q["Query API
(internal/api)"]
M["Merge results
(federate.go)"]
Q -->|fan out| N1
Q -->|fan out| N2
Q -->|fan out| N3
N1 -->|results| M
N2 -->|results| M
N3 -->|results| M
M -->|single result set| D["Dashboard
+ Prometheus
remote-read"]
```
A Prometheus remote-read endpoint exposes the same data, so you can point your own Grafana at it.
Alerting and detection sit on top of the metrics and log layer:
* Alerting rules (threshold, crashloop, certificate expiry, and more) delivered over 18 notification kinds, from Slack, Discord, email, and Telegram to PagerDuty, ntfy, a generic webhook, and browser push
* Crashloop detection that surfaces the last 200 lines of a failing container's logs directly in the UI
## Secrets
Package: `internal/secrets` (see `internal/secrets/manager.go`, `internal/secrets/dek.go`).
Secrets use envelope encryption: a per-app data encryption key, wrapped by a master key the control plane holds. Agents receive decrypted environment variables only at container-create time and never write them to disk. The crypto primitives come from `filippo.io/age`.
## Frontend
Directory: `web/` (React, Vite, TypeScript, Tailwind, TanStack Router and Query).
The frontend builds to static assets and is embedded into the control plane binary via `embed.FS`, so there's no separate Node process to run in production.
Route-level code splitting keeps bundles small. The log viewer's bundle, for example, stays out of the initial dashboard load.
Live build logs and app log tailing use server-sent events (SSE) instead of websockets, because SSE reconnects cleanly through proxies.
## See also
* [Comparison](/comparison): how this architecture compares to Coolify, Dokploy, and others
* [Security overview](/security): how secrets, tokens, and TLS fit into this design
* [Feature status](/feature-status): maturity of each layer
* [ADRs](https://github.com/glincker/levelrail/tree/main/adr): the decisions behind this design
# Attention center
Source: https://docs.levelrail.com/attention-center
How the dashboard surfaces what waits on an operator: CLI login approvals, deploy approvals, expiring tokens, failed data copies, overdue backups and renewals.
Some states block on a person and used to be visible only if you went looking. The attention center puts them in one place: the banner and bell in the dashboard header, the [Status page](/status-page), `levelrail-cli attention`, and the MCP `get_attention` tool.
## What shows up
| Item | Where it comes from | What to do |
| - | - | - |
| CLI login waiting | `levelrail-cli auth login --device` | Compare the code, then approve or deny |
| CLI login expired or denied | A login nobody approved in time, or that was denied | Start over from the terminal, or dismiss |
| Deploy approval pending | A deploy into a protected environment | Open Approvals |
| API token expiring or expired | A token with an expiry inside the window | Create a replacement and revoke the old one |
| Data copy failed or stalled | The latest live data copy into a managed database | Read the plain-language cause, fix it and retry |
| Backup overdue | A scheduled backup later than its schedule allows | Open the database and run a backup |
| Invitations waiting | Pending or expired, unaccepted invitations | Open Users and resend or revoke |
| Certificate renewal stalled | A certificate that should have renewed by now, with the CA's last error | Open Domains, read the CA error |
| Node needs to be re-enrolled | A node enrolled by an early beta whose key was made by the control plane. It works until its certificate expires, then cannot renew. Severity follows the time left, using `APP_NODE_CERT_EXPIRY_WARNING` and `APP_NODE_CERT_EXPIRY_CRITICAL`. It clears when the node re-enrolls. | Open the node and use **Re-enroll** (the item links straight to it) |
| Agent certificate renewal failing | A node certificate in its warning or critical window. The item carries the agent's last renewal error (shortened, never key material) and the next retry time. | Read the error, fix the node or network, or renew now |
| Node joined but never connected | An enrolled node whose agent never reported in | Check the agent logs and network path |
Existing items (failing apps, failed deploys, offline nodes, disk pressure, expiring certificates, doctor warnings, updates) appear in the same list. Items that wait on a person sort ahead of other warnings, critical items always come first, and informational items (resolved logins) come last.
Every item has a stable id, a severity (`critical`, `warning` or `info`), a one-line title, a next action and a dashboard link. A token only receives the items its abilities allow, so a read-only token gets a shorter list instead of an error.
## Approving a CLI login
A pending login raises a banner on every page of the dashboard with the code, where the request came from, the device name, and a countdown. The banner cannot be dismissed. It goes away when the login is approved, denied, or expires (10 minutes by default, `APP_DEVICE_CODE_TTL`).
1. Look at the terminal that ran `levelrail-cli auth login --device`.
2. Click **Review and approve**. The code is shown large.
3. Click **The codes match, approve** only if it is exactly the code in your terminal.
Approval is never automatic and never a single click. If the requesting IP address differs from the IP your browser session uses, the banner and dialog show a warning. Only approve a login you started.
Only a person signed in to the dashboard can approve or deny. API tokens, the CLI and the MCP server cannot, so an AI agent that starts a login has to ask you to approve it.
## Expired and denied logins
When a login expires or is denied it stays visible for 24 hours (`APP_ATTENTION_RESOLVED_WINDOW`) as an informational strip under the banner, in the bell and on the Status page. Each shows its state (waiting, expired, denied or approved), the requesting IP, the device name and how long ago it happened.
* **Start over** shows the command to run in your terminal to request a new login.
* **View audit entries** opens the audit log filtered to that one request.
* The cross **dismisses** the item for you only. Dismissal is stored on the server, so it survives a reload and applies on your other devices. It never edits the audit log, and a new request is a new item that appears again.
A waiting login cannot be dismissed. Approved logins need no action and appear only in the audit log.
## Audit trail
Every state change writes exactly one audit entry with a stable action name, shown under **Settings > Audit log** with the **Device login** filter (`?action=device_login`):
| Action | Written when |
| - | - |
| `device_login.approved` | A signed-in operator approves |
| `device_login.denied` | A signed-in operator denies |
| `device_login.expired` | A pending request passes its time limit (actor `system`) |
| `device_login.dismissed` | A user dismisses an expired or denied item |
A background sweeper records expiry whether or not the CLI ever polls again. It is idempotent and safe across restarts: an expiry is claimed once in the same transaction that writes its audit entry. Entries carry the request id, never the code. Set the sweep interval with `APP_DEVICE_EXPIRY_SWEEP_INTERVAL` (default 30s) and how long resolved requests are kept with `APP_DEVICE_HISTORY_RETENTION` (default 30 days; audit entries follow the audit log retention instead).
## Sign-in codes and new browsers
When someone asks for a sign-in code for your account, or signs in with your password from a browser you have not trusted while you are signed in here, a banner appears on every page and the attention center lists one `login_code` or `login_approval` item for your account, with a count and the newest requester's IP address, browser and time. Items and the banner never carry the code: **Show code** fetches it on demand, and each reveal is audited. Approve or deny a new browser from the banner or from **Settings > Security**; approving asks you to type the number the waiting browser shows. The same requests are available from the terminal with `levelrail-cli auth code`. See [Identity and access](/identity-and-access#sign-in-with-a-code).
## Running the CLI through a tunnel or proxy
The link printed by `auth login --device` is built from the address the CLI actually reached, so it works behind an SSH tunnel, a reverse proxy, or a custom port. A few rules apply.
* `APP_DASHBOARD_URL`, when set, always wins.
* Otherwise the request's `Host` header is used.
* `X-Forwarded-Host` is honored only when the request arrives from a loopback address or one listed in `APP_INGRESS_TRUSTED_PROXIES`.
## For AI agents
`levelrail-cli auth login --device --json` prints one JSON line to stdout before it blocks:
```json theme={"dark"}
{"event":"device_login_pending","verification_url":"http://127.0.0.1:28080/settings/cli-access?user_code=ABCD-1234","user_code":"ABCD-1234","expires_in":600,"expires_at":"2026-10-09T12:10:00Z"}
```
Relay `verification_url` and `user_code` to the person. The final token resource follows on stdout once they approve.
## Get notified outside the dashboard
A channel can opt in to a notice when a CLI login is waiting, and separately when a pending login expires. Both are off by default.
```
levelrail-cli channels update --name NAME --kind slack --notify-url URL --notify-device-login --notify-device-login-expired
```
You can also toggle both when editing a channel in the dashboard. A notice contains a link only, never the code. The link needs `APP_DASHBOARD_URL` (or `APP_PUBLIC_HOST`) to be set, and at most one notice of each kind is sent per minute.
## Tuning
| Variable | Default | Meaning |
| - | - | - |
| `APP_DEVICE_CODE_TTL` | `10m` | How long a login request stays approvable |
| `APP_DEVICE_EXPIRY_SWEEP_INTERVAL` | `30s` | How often expiry is recorded |
| `APP_ATTENTION_RESOLVED_WINDOW` | `24h` | How long expired and denied logins stay visible |
| `APP_DEVICE_HISTORY_RETENTION` | `720h` | How long resolved requests are kept |
| `APP_ATTENTION_TOKEN_WINDOW` | `168h` | How far ahead token expiry is flagged, and how long after |
| `APP_ATTENTION_BACKUP_GRACE` | `6h` | Slack past a backup's expected interval before it is overdue |
| `APP_MAINTENANCE_ATTENTION_AFTER` | `4h` | How long a domain may stay in maintenance before it is raised as an item. Only domains of apps the caller can read are raised; a domain with no app needs the root ability. Maintenance mutes only down, DNS, certificate and request error or latency alerts, so CPU, memory, disk, restart and crash alerts still surface here |
| `APP_ATTENTION_SHORT_LIVED_CERT` | `168h` | Certificates valid for no longer than this are auto-renewed internal issuance and never raise an expiry item |
| `APP_ATTENTION_COPY_FAILURE_DAYS` | `14` | Days a failed data copy stays an incident before it moves to history |
## Incidents and recommendations
Every item has one of two classes, decided on the server. An incident is broken or about to break and needs action: failed deploys, crashloops, a real certificate that expired or whose renewal stalled, disk pressure, overdue backups, offline nodes, active sign-in threats. Only incidents appear in the dashboard's Needs attention list and in the header count.
A recommendation is setup or hardening advice: control plane [disaster recovery](/disaster-recovery) is off, the Docker [API guard](/docker-access) could move from audit to enforce, a new release is available, an agent is out of date, a database release is past end of life. These wait in the notification centre under Pending setup and hardening, where each can be snoozed for a day or dismissed. The dashboard shows them as a single line with a count.
Certificate expiry only becomes an incident for a certificate that is routed to an app or site, is not issued by the internal CA, and whose TLS is not terminated by a proxy. An auto-renewing ACME certificate counts only once it expired or its renewal stalled; an uploaded certificate counts from the start of the warning window.
## Failed data copies
A failed copy is an incident only while it is the latest attempt for that database. It drops off by itself when a restore into that database succeeded after it, when a newer attempt is running or has been verified, or when it is older than `APP_ATTENTION_COPY_FAILURE_DAYS`.
The item states the cause in one sentence with a fix hint, for example "The source database did not answer on 72.61.8.70:5432 in time. Check that it is reachable from this server (firewall, allowed IPs) and retry." Covered shapes: connection timeout, connection refused, host name not found, authentication failed, database does not exist, SSL required or refused, permission denied, out of disk space, target database not running, and a verification mismatch. The raw error is under **Technical details** in the dashboard and behind `--verbose` in `levelrail-cli migrate db-status`. It is never the default view, and passwords and connection strings are redacted before it leaves the server.
Each failed attempt can be snoozed for a day (remembered in this browser) or dismissed (remembered for your account). Both apply to that attempt only: a new failed attempt shows again.
## Security notes
* Starting a login is unauthenticated and rate limited per IP (`APP_DEVICE_START_RATE_PER_MINUTE`, default 6, `0` disables).
* Approve, deny and dismiss are session-only and pass the cross-origin check. The read endpoints (`/api/v1/auth/device/activity`, `/api/v1/attention/feed`, `/api/v1/auth/device/pending-summary`) never include a code.
* Codes never appear in logs, audit entries, notices or MCP output. The MCP server stays read-only and has no approve, deny or dismiss tool.
# Sign-in engine
Source: https://docs.levelrail.com/auth-engine
How sign-in, API tokens, two-factor, passkeys and OAuth are served, and the settings that tune them.
Sign-in, sessions, API tokens, device login, two-factor codes, passkeys and OAuth all run on one auth library built into the control plane. On a new install there is nothing to enable or migrate. The first account is created from the setup token and goes straight into the library.
Check what is available with `levelrail-cli auth-engine status` (root only) or on Settings > Security. It reports the library version and whether two-factor, passkeys and OAuth are available.
## Requirements
| Feature | Needs |
| - | - |
| Two-factor codes | The control plane master key |
| Passkeys | A dashboard URL set to a domain (or `APP_AUTH_ENGINE_WEBAUTHN_RP_ID`). An IP address cannot be a relying party. Changing the URL needs a restart |
| OAuth and OIDC sign-in | The master key, and `APP_AUTH_ENGINE_BASE_URL` set to the public dashboard URL |
## Settings
All optional.
| Variable | Default | Purpose |
| - | - | - |
| `APP_AUTH_ENGINE_BASE_URL` | derived from the listen address | Public URL used in generated links and OAuth callbacks |
| `APP_AUTH_ENGINE_PATH_PREFIX` | `/api/v1/auth-lib` | Route prefix of the library's own routes |
| `APP_AUTH_ENGINE_PASSWORD_MIN_LENGTH` | `8` | Minimum password length |
| `APP_AUTH_ENGINE_DEVICE_POLL_INTERVAL` | `5s` | Device login poll interval |
| `APP_DEVICE_TOKEN_TTL_DAYS` | `30` | Lifetime of a device login token |
| `APP_DEVICE_TOKEN_ALLOW_ROOT` | `false` | Let a root approver mint a root device token |
| `APP_SESSION_TTL` | `24h` | Absolute session lifetime |
| `APP_AUTH_ENGINE_SESSION_IDLE_TIMEOUT` | `0` (off) | End a session unused for this long |
| `APP_AUTH_ENGINE_SESSION_LINK_TTL` | `2m` | How long a session link can be exchanged |
| `APP_AUTH_ENGINE_LOGIN_GRACE_FAILURES` | `3` | Free failed logins per IP and username before backoff |
| `APP_AUTH_ENGINE_LOGIN_BASE_DELAY` | `1s` | First backoff delay, doubled per further failure |
| `APP_AUTH_ENGINE_LOGIN_MAX_DELAY` | `15m` | Backoff cap |
| `APP_AUTH_ENGINE_LOGIN_RESET_AFTER` | `15m` | Forget failures after this idle time |
| `APP_AUTH_ENGINE_LOGIN_USER_MAX_FAILURES` | `10` | Consecutive failures on one account that lock it |
| `APP_AUTH_ENGINE_LOGIN_USER_LOCKOUT` | `15m` | How long that lock lasts |
| `APP_AUTH_ENGINE_RATE_LIMIT_PER_IP` | `60` | Credential requests per minute per IP |
| `APP_AUTH_ENGINE_RATE_LIMIT_PER_EMAIL` | `60` | Credential requests per minute per username |
| `APP_AUTH_ENGINE_STREAM_WATCH_INTERVAL` | `5s` | How often a live stream re-checks its session |
| `APP_AUTH_ENGINE_WEBAUTHN_RP_ID` | host of the dashboard URL | Passkey relying party ID |
| `APP_AUTH_ENGINE_WEBAUTHN_ORIGINS` | the dashboard URL origin | Extra allowed passkey origins, comma separated |
| `APP_AUTH_ENGINE_WEBAUTHN_REQUIRE_UV` | `false` | Demand PIN or biometric on every passkey use |
| `APP_AUTH_ENGINE_WEBAUTHN_CLONE_WARNING` | `reject` | `reject` refuses a sign-in whose counter went backwards, `flag` allows it and audits |
| `APP_AUTH_ENGINE_MFA_MAX_FAILURES` | `5` | Wrong codes per user before a lockout |
| `APP_AUTH_ENGINE_MFA_LOCKOUT` | `15m` | How long that lockout lasts |
| `APP_AUTH_ENGINE_OAUTH_ALLOWED_HOSTS` | empty | Extra hosts allowed in an OAuth redirect URI once a dashboard URL is set |
| `APP_AUTH_ENGINE_OAUTH_PROVIDER_TTL` | `30s` | How long a resolved OAuth provider is cached. Saving provider settings drops it at once |
## Upgrading from a release before the auth library
The auth library replaced the old built-in sign-in, and that change breaks existing installs (the platform had no production users when it landed). After upgrading from `v0.2.0-beta.18` or earlier:
* Everyone is signed out. Sign in again with your existing username and password: accounts and password hashes carry over.
* API tokens created before the upgrade stop working (they answer 401). Create new ones in **Settings, CLI Access**, or run `levelrail-cli auth login --device`, and update CI secrets.
* Two-factor codes, passkeys and recovery codes are not carried over. Re-enroll them after signing in.
Take a backup of the data directory before upgrading, as the installer's upgrade notes recommend. Each start also writes a pre-migration database snapshot you can restore with `levelrail restore-db`.
## Things to know
* API tokens carry the `tk_` style prefix of the product. A token's effective abilities never exceed its owner's current abilities.
* A TOTP code can be used once. Repeated wrong codes lock the user out, counted across devices, and answer 429 with `Retry-After`.
* Recovery codes look like `XXXX-XXXX-XXXX-XXXX` and are accepted with or without hyphens and in any case.
* Sign-in is case-insensitive on the username. Usernames that are not an email address (for example `admin`) are still accepted at first-run and by invites.
* OAuth callback URL: register `https:///api/v1/auth/oauth//callback` at each identity provider. Sign-in and profile linking share it, and Levelrail tells them apart by the state value. Once a dashboard URL is set, redirect hosts are limited to it and the base URL host, plus any in `APP_AUTH_ENGINE_OAUTH_ALLOWED_HOSTS`. The redirect URI is always built from the dashboard URL (or `APP_DASHBOARD_URL`). With neither set, the request's `Host` and `X-Forwarded-Proto` are used only from a loopback peer or one listed in `APP_INGRESS_TRUSTED_PROXIES`; any other caller gets a `400` until a dashboard URL is set.
* An existing account is linked on OAuth sign-in only when the provider reports the email as verified. A provider's allowed email domain gates new accounts only.
* `recover-admin` sets the password through the library, which also ends that account's sessions and clears its lockout. No restart is needed. Locked out another way? See [Emergency access](/emergency-access).
## Troubleshooting
### Passkeys not available, getting "Passkey registration is not available"
Likely cause: The control plane is running on an IP address instead of a domain name, or `APP_AUTH_ENGINE_WEBAUTHN_RP_ID` is not set.
How to confirm: Run `levelrail-cli auth-engine status` and check the "Passkeys" line. If it says "not available", passkeys are disabled. Check what URL is set: Dashboard Settings > Security or run `curl https:///api/v1/brand`.
Fix: Set your dashboard to a domain name (not an IP), or set `APP_AUTH_ENGINE_WEBAUTHN_RP_ID` to your domain and restart the control plane.
Still stuck: See [Configuration](/environment-variables) for how to set environment variables and restart.
### Two-factor codes not working or "Enrollment unavailable"
Likely cause: The master key is not configured. Two-factor, passkeys and OAuth all need the control plane's encryption master key to function.
How to confirm: Run `levelrail-cli auth-engine status` and check whether "2FA" and "OAuth" are available. Both should show "available (requires master key)". If the master key is missing, they will show "not available".
Fix: Set `APP_MASTER_KEY` to the master key value and restart the control plane. See [Master key rotation](/master-key-rotation) for how to set or recover the master key.
Still stuck: Read [Master key](/master-key-rotation) if you lost the master key.
### OAuth callback failing with "redirect\_uri\_mismatch"
Likely cause: The OAuth redirect URI registered at the provider does not match what the control plane is sending. The platform builds the URI from the dashboard URL, and the host must match exactly (including `www.`).
How to confirm: Check your OAuth provider's console for which URIs are registered. On the control plane, run `curl https:///api/v1/brand` and note the exact domain in "domain". Register `https:///api/v1/auth/oauth//callback` at the provider.
Fix: Update the registered redirect URI at your OAuth provider to exactly match `https:///api/v1/auth/oauth//callback`. If you need to use a different hostname, set `APP_AUTH_ENGINE_OAUTH_ALLOWED_HOSTS` to comma-separated allowed hosts.
Still stuck: See [SSO and SAML](/sso-saml) for identity provider setup.
### API token returns 401 "Unauthorized" after an upgrade
Likely cause: You upgraded from v0.2.0-beta.18 or earlier, and API tokens created before the upgrade are no longer valid (the auth library replaced the old sign-in system).
How to confirm: Check when the token was created. If it was before the upgrade, it is obsolete. Run `levelrail-cli auth login --device` to mint a new token.
Fix: Create a new API token in Settings > CLI Access (dashboard) or by running `levelrail-cli auth login --device`, then update your CI or scripts to use the new token.
Still stuck: See [API reference](/api-reference) for the API surface.
# Backup targets, registry credentials, and app volume backups
Source: https://docs.levelrail.com/backups-and-storage
Configure backup targets, registry credentials, and automated backup schedules with retention policies.
![Amazon S3]( "Amazon S3")
![MinIO]( "MinIO")
This guide covers three related resources:
1. Backup targets: S3-compatible buckets where database and app volume backups are stored.
2. Registry credentials: authentication for pulling private images from external registries.
3. The built-in registry: Levelrail's own `registry:2` container for pushing and caching images locally.
All three share one pattern: an external system that needs a working credential, which you test on demand instead of finding out it is broken mid-deploy or mid-backup. For the database side of backups (restore, point-in-time recovery, verification), see [Managing databases](/managing-databases). For backing up the control plane itself, see [Control plane backup](/control-plane-backup) and [Disaster recovery](/disaster-recovery).
## Backup lifecycle at a glance
```mermaid theme={"dark"}
graph LR
A["Create
backup target"] --> B["Test
connection"]
B --> C["Define
schedule"]
C --> D["Backup runs
on schedule"]
D --> E["Retention
pruning"]
E --> F["Restore
if needed"]
D -.->|manual trigger| F
```
## Quick start
```bash theme={"dark"}
levelrail-cli backup-targets create \
--name "primary-r2" \
--provider r2 \
--endpoint https://.r2.cloudflarestorage.com \
--bucket levelrail-backups \
--access-key-id AKIA... \
--secret-access-key ...
```
The response never echoes the credentials back. `--provider` accepts `aws`, `r2`, or `custom`. `--endpoint` is required for `r2` and `custom`, and `aws` must omit it (AWS resolves its own endpoint per region). For Backblaze B2, Wasabi, or MinIO, use [Object storage](/object-storage) to connect the bucket as a storage destination: the same record shows up under backup targets.
```bash theme={"dark"}
levelrail-cli backup-targets test bkt_9f2a1c...
```
The command exits non-zero with a specific message ("authentication rejected", "bucket not found") if the credential does not work, and succeeds silently if it does.
```bash theme={"dark"}
levelrail-cli backups schedule set my-postgres \
--target bkt_9f2a1c... \
--cron "0 3 * * *" \
--retain 7 \
--retain-days 30
```
```bash theme={"dark"}
levelrail-cli app-volume-backups schedule set my-app uploads \
--target bkt_9f2a1c... \
--cron "0 4 * * *" \
--retain 14
```
## Why a backup target is its own resource
A backup target is not a field on a database or volume. It is a separate resource that any number of databases and app volumes reference by ID. You connect a bucket once in **Settings, Backup targets**, every backup schedule references it by `target_id`, and rotating a leaked access key happens in one place.
### Credentials are write-only
Credentials never round-trip through the API.
* `access_key_id` and `secret_access_key` are accepted only in create and update request bodies.
* They are written to the secrets store before the backup target record is saved, so a failed save never orphans a credential.
* `GET` and `PUT` responses never echo credential fields, and error responses never include them.
Registry credentials follow the same pattern: only the `password` field is accepted, it is written to the secrets store first, and it is never echoed back. When you reference a registry credential in `app.yaml` by name, the deploy authenticates without the password being stored in the file.
Every write endpoint on this page returns `501` if the control plane has no master key configured, because none of these resources can hold a working credential without one.
## Test a connection before it is needed
Both resources expose a `POST .../test` endpoint that validates the credential without doing the real operation.
![Levelrail Backups page on a fresh install, prompting you to add a backup target]()
`POST /api/v1/backup-targets/{id}/test` calls `HeadBucket` against the bucket. Nothing is uploaded or deleted.
* `401` or `403`: authentication rejected. Check the access key ID and secret access key.
* `404`: bucket not found. Check the bucket name, region, and endpoint.
* Anything else: the backup target could not be reached.
CLI: `levelrail-cli backup-targets test `. Dashboard: the **Test connection** button on each row of the Backup targets table.
`HeadBucket` proves the bucket exists and can be reached, but not that the key can write or delete, which backups and pruning need. A storage destination's connection test (`levelrail-cli storage test `) does a full write, read, and delete round trip, see [Object storage](/object-storage#what-the-connection-test-does).
`POST /api/v1/registry-credentials/{id}/test` authenticates the stored username and password against the registry host the same way a real image pull does. No image is pulled.
* Unauthorized response: authentication rejected by the registry. Check the username and password.
* Anything else: the registry could not be reached.
CLI: `levelrail-cli registry-credentials test `. Dashboard: the **Test connection** button on each row of the Registry credentials table.
Better to find a stale credential here than from a failed 3am backup or a deploy that dies mid-pull.
## Database and app volume backups
A backup target does not know what it is backing up: databases and app volumes reference the same `target_id`.
| Resource | Trigger | History | Schedule |
| - | - | - | - |
| Database | `POST /api/v1/databases/{name}/backups` | `GET /api/v1/databases/{name}/backups` | `PUT`/`DELETE /api/v1/databases/{name}/backup-schedule` |
| App volume | `POST /api/v1/apps/{name}/volumes/{volume}/backups` | `GET /api/v1/apps/{name}/volumes/{volume}/backups` | `PUT`/`DELETE /api/v1/apps/{name}/volumes/{volume}/backup-schedule` |
Both trigger endpoints validate `target_id` against a real backup target, create a backup history record, start the dump-and-upload (or volume-tar-and-upload) in the background, and return `202 Accepted` immediately. Poll the history endpoint to see the result: status starts at `running`.
A database dump and a volume archive both stream from their source to the bucket and never touch local disk. The one local write is the backup's own history row, in the control plane's SQLite database under `APP_DATA_DIR`. Before starting, the control plane checks free space there and fails fast with `507` instead of failing confusingly partway through. The floor is `APP_MIN_BACKUP_DISK_MB` (default `256`); an unconfigured or unreadable data directory skips the check.
### Schedules
Database schedules are stored on the database itself. App volume schedules are stored per named volume, because a service can declare several named volumes in `app.yaml` and there is no single "app" to attach one schedule to. Both use the same cron validation (a bad cron string returns `400` at request time), the same retention fields, and the same check that the target exists before saving.
On the dashboard, a database's schedule form is on its Overview page and an app volume's is on the app's Volumes tab. Backup targets themselves are configured account-wide under **Settings, Backup targets**.
### Volume backup commands
App volume backups have the same lifecycle as database backups:
```bash theme={"dark"}
levelrail-cli app-volume-backups trigger --target ID
levelrail-cli app-volume-backups list
levelrail-cli app-volume-backups verify --backup ID
levelrail-cli app-volume-backups restore --backup ID --confirm APP/VOLUME
levelrail-cli app-volume-backups restore-as-new --backup ID
```
### Compression, encryption and consistency
New app volume backups are streamed as tar, compressed with zstd and encrypted with age on the control plane before upload, so the bucket only ever holds ciphertext. Each backup records its encoding, so backups from before this feature stay restorable. The key, retention buckets, consistency hooks and size and time budgets are covered in [Prove your backups work](/prove-your-backups-work).
Database dumps are not re-encoded: they stream straight from the dump command as before.
### Restore to a new volume, app or node
`POST /api/v1/apps/{name}/volumes/{volume}/restore-to` restores a volume backup into a new volume and never writes to an existing one. It takes an optional `target_app` (names the volume for that app so deploying it mounts the data), an optional `node_id`, and an optional `new_volume_name`. On the CLI:
```bash theme={"dark"}
levelrail-cli backups volumes restore --backup ID [--target-app APP] [--node NODE] [--new-volume NAME]
```
### Instance-wide backup visibility
`GET /api/v1/backups` (newest first, cursor-paginated with `?limit=&before=`) lists backups across every database and app volume. Each entry carries `resource_kind` (`database` or `volume`) and identity fields (`database_name`, or `service_name` and `volume_name`).
On the CLI the attempt history is `levelrail-cli backups list-all [--limit N] [--before RFC3339]` and the health view is `levelrail-cli backups health`.
### Coverage: what is and is not protected
The top-level **Backups** page leads with coverage, not history. It answers "is everything backed up, and has a restore been proven?" at a glance:
* A summary: how many resources are protected, the last verified restore, the next run and the storage used.
* One row for every database and every app volume, protected or not, each with a state and one primary action.
* A compact card for encryption and bucket safety, a small timeline of restore drills, an **Orphaned data** panel, and a collapsed **Recent activity** section holding the attempt history.
Each resource is in exactly one state:
| State | Meaning | Primary action |
| - | - | - |
| Protected and verified | Scheduled, has a backup, and a restore drill passed | Restore |
| Protected, not verified | Scheduled and backed up, but no restore has been proven | Restore (and Prove restore) |
| Scheduled, never ran | A schedule exists but no backup has completed | Back up now |
| Backups failing | Every attempt failed, or the last drill failed | Back up now |
| Not protected | No schedule or no target. A backup taken by hand does not count | Protect |
A resource counts as protected only when it is scheduled and has a successful backup. [Preview environments](/previews) and canaries are marked **Scratch** and left out of the totals.
**Protect everything** (`POST /api/v1/backups/protect-all`) schedules every unprotected resource in one step. You pick a destination (or connect one inline from a provider preset: Cloudflare R2, Backblaze B2, Wasabi, AWS S3, MinIO, or local disk), a schedule, and how many backups to keep. The dialog previews the plan first, and nothing changes until you apply. Scratch resources are off unless you opt in. The same flow on the CLI:
```bash theme={"dark"}
levelrail-cli backups coverage [--unprotected]
levelrail-cli backups protect-all --target ID # prints the plan, changes nothing
levelrail-cli backups protect-all --target ID --apply [--schedule "0 3 * * *"] [--retain 7] [--retain-days 30]
```
The default schedule and retention come from `APP_BACKUP_DEFAULT_SCHEDULE` (default `0 3 * * *`), `APP_BACKUP_DEFAULT_RETAIN` (default 7) and `APP_BACKUP_DEFAULT_RETAIN_DAYS` (default 30). They are suggestions: a database without a schedule shows them on its own Backups card once a destination exists, and applies them only when you click. The dashboard also shows a recommendation when resources are not backed up.
The **Local disk** preset points at an S3 server on this machine. Backups stored on the machine they protect are lost with it. It is fine to try things out and wrong for anything you care about.
#### The encryption key
The card shows whether new volume backups are encrypted. The public key is never printed on the page: use **Copy public key** in the disclosure. The private key must be stored off this server, or encrypted backups cannot be restored after a disaster. Until you confirm that you did (`POST /api/v1/backups/encryption-key/ack`), the card keeps a warning. The confirmation is a statement from you, not something the control plane can verify, and it is tied to the public key, so rotating the key asks again.
### Orphaned data
Deleting a database or app does not delete its Docker volumes, so a mistaken delete never destroys data. Volumes this control plane created whose database or app no longer exists are listed under **Orphaned data** with size, age, which resource they belonged to, and the last backup of it (a volume with no backup is flagged).
* **Inspect** shows the volume, its role (data, certificates, WAL archive), its creation time and its last backup.
* **Reclaim** deletes the volume after you type its name. The server re-checks that it is still orphaned and not mounted, so a volume an app started using again is refused.
* Nothing is ever deleted automatically.
* When orphaned data exceeds `APP_ATTENTION_ORPHAN_BYTES` (default 1 GiB), a recommendation appears on the dashboard.
To destroy the data together with the resource, tick **Also delete the data** in the database or app delete dialog, or pass `--delete-data --confirm NAME` on `levelrail-cli databases delete` and `levelrail-cli apps delete`. The API takes `?delete_data=true&confirm=NAME` and needs `write:sensitive`. It only works for resources on the control plane's own node, and the volumes are removed after the containers are gone, so a teardown that has not finished leaves them listed as orphaned data instead.
```bash theme={"dark"}
levelrail-cli backups orphans list
levelrail-cli backups orphans reclaim db-orders-data --confirm db-orders-data
```
Coverage, protect-all and orphaned data are instance wide. Members of non-default teams cannot reach them.
## Retention
Retention has two independent dimensions, both optional:
* `retain` (count): keep the newest N successful backups by `started_at`.
* `retain_days` (age): keep only backups less than N days old.
They combine, so a backup is deleted when either rule would remove it. With `retain: 7` and `retain_days: 30`, you keep at most the 7 newest backups, and never anything older than 30 days. `0` disables a dimension, and `0` for both disables pruning.
Only succeeded backups are ever pruned. Running and failed attempts are kept regardless of age or count, since you may need a failed attempt's logs.
Pruning runs after every scheduled backup that succeeds, not after manual ones. A pruning failure is logged and does not fail the backup, which is already durably recorded. The dashboard's schedule form defaults a new database's "keep last N backups" to 7 and "delete backups older than (days)" to 0 (no age limit).
## Delete one backup on demand
Retention removes old backups automatically. To delete one specific backup right now without touching the schedule:
* CLI: `levelrail-cli backups delete ` or `levelrail-cli app-volume-backups delete `
* API: `DELETE /api/v1/databases/{name}/backups/{historyId}` or `DELETE /api/v1/apps/{name}/volumes/{volume}/backups/{historyId}`
* Dashboard: the Delete action on every non-running backup row, behind a confirm dialog.
Naming the exact backup ID is the confirmation, so there is no `--confirm` flag. The stored object is removed first (best effort), then the history row. If the object is already gone, the target was deleted, or storage returns an error, the failure is logged and the row is removed anyway, so an explicit delete is never stuck in the history. A backup that is still `running` cannot be deleted (`409`). The endpoint needs `write:sensitive`.
## Alert on a missing or failing backup
A `restore_drill_failed` rule fires while any resource's latest restore drill is failed. A `backup_missing` alert rule watches each database's and app volume's backup schedule. It fires when the last successful backup trails the expected interval by more than the rule's `for_duration` (default 6 hours). It reads the same schedule and history rows the scheduler uses, so it catches both a schedule that silently stopped running and repeated failures with no recent success. It notifies through your normal alert channels. See [Observability](/observability#alert-rules) for the rule table and setup.
## Registry credentials
Registry credentials authenticate to external registries such as Docker Hub or ghcr.io, so deploys can pull private images.
```bash theme={"dark"}
levelrail-cli registry-credentials create \
--name ghcr-private \
--registry-host ghcr.io \
--username you \
--password ghp_...
```
`--expires-at RFC3339` records an expiry you already know, such as a GitHub token's.
```yaml theme={"dark"}
services:
web:
build:
type: image
image: ghcr.io/you/private-app:latest
registryCredential: ghcr-private
```
```bash theme={"dark"}
levelrail-cli registry-credentials test
```
`expires_at` is informational: it drives a healthy, expiring soon, or expired badge (like TLS certificates), and nothing blocks a deploy from using an expired credential. There is no automatic rotation.
## The built-in container registry
The built-in registry is Levelrail's own `registry:2` container, a local image distribution point with no external dependency. It is separate from registry credentials above.
### Enable it
```bash theme={"dark"}
levelrail-cli registry enable --host registry.internal.example
```
This calls `PUT /api/v1/settings/registry` with `enabled: true` and the host. It generates the fixed username `levelrail` and a random password, and returns the password once. If you lose it, disable and re-enable to get a new one. Later calls (changing the host, re-enabling) leave existing credentials untouched.
```text theme={"dark"}
enabled: true
host: registry.internal.example
username: levelrail
has_credentials: true
status: running
password:
This password is shown once and never again. Store it now, e.g.:
docker login registry.internal.example -u levelrail
```
`levelrail-cli registry disable` (`DELETE /api/v1/settings/registry`) clears the generated credentials and is idempotent.
### Browse repositories and tags
Repositories and tags come from the running registry's HTTP API v2 catalog, authenticated server-side, so the caller never needs the password. On the dashboard, the "Existing image" picker when creating an app uses this. On the CLI:
```bash theme={"dark"}
levelrail-cli registry repositories
levelrail-cli registry tags --repository NAME
levelrail-cli registry-credentials repositories
levelrail-cli registry-credentials tags
```
## What has been tested, and limits
These ran end to end against a real S3-compatible endpoint (SeaweedFS) and real database containers:
| Check | Result |
| - | - |
| Backup and restore-as-new for Postgres, MySQL, MariaDB, MongoDB, and Redis (TLS on) | Row and document counts match the source |
| Backup, verify, and restore-as-new for ClickHouse, KeyDB, and Dragonfly (hyphenated names) | Rows and keys match the source |
| Flip one byte of a stored backup, then verify | Verification fails with a checksum mismatch |
| Restore from that corrupted object | Refused before the database is touched |
| Postgres restore of a dump with a failing statement | Fails and leaves the existing data untouched |
| Postgres point-in-time restore, including after deleting both the data volume and the WAL archive volume | Returns exactly the data from before the target time, from the copy shipped to the bucket |
| App volume backup and restore-as-new with a directory owned by uid 1234:5678 (mode 750), a 640 file with an old mtime, and a symlink | Owners, modes, mtimes, and the symlink are identical |
| Create a database with the name of a deleted one | Refused until you choose to reuse or discard the old data |
| Free-space floor set absurdly high | Manual triggers return `507`; a major upgrade fails in its first phase and the database is never stopped |
Limits to know about:
* A backup is a logical dump (`pg_dump`, `mysqldump`, `mongodump`, an RDB snapshot). It is consistent per database but not a physical copy. For Postgres, use point-in-time restore when you need to recover to an exact second.
* Verification catches a damaged or truncated object. It does not prove the dump restores cleanly. Restore drills do: they restore into a scratch resource, validate it and destroy it, automatically once a week by default. See [Prove your backups work](/prove-your-backups-work).
* Deleting a database keeps its data volume and your backups. See [Managing databases](/managing-databases#stop-start-and-delete).
* The free-space floor protects the control plane's data directory. Nothing watches the Docker host's own disk for a major upgrade's snapshot copy: a copy that runs out of space fails the upgrade before it changes anything.
* WAL for point-in-time restore is shipped to the backup target only for databases on the control plane's own node.
* A storage endpoint must resolve to a public address unless `APP_NOTIFY_ALLOW_PRIVATE_NETWORKS=true` is set, which a MinIO on the same private network needs.
* Deleting a backup target leaves its stored access key and secret in the secrets store, unreferenced and unreachable through the API but not erased at rest. Registry credentials behave the same way.
## API reference
| Method | Path | Ability |
| - | - | - |
| `GET` | `/api/v1/backup-targets` | `read` |
| `POST` | `/api/v1/backup-targets` | `write:sensitive` |
| `GET` | `/api/v1/backup-targets/{id}` | `read` |
| `PUT` | `/api/v1/backup-targets/{id}` | `write:sensitive` |
| `DELETE` | `/api/v1/backup-targets/{id}` | `write:sensitive` |
| `POST` | `/api/v1/backup-targets/{id}/test` | `write:sensitive` |
| `GET` | `/api/v1/registry-credentials` | `read` |
| `POST` | `/api/v1/registry-credentials` | `write:sensitive` |
| `GET` | `/api/v1/registry-credentials/{id}` | `read` |
| `PUT` | `/api/v1/registry-credentials/{id}` | `write:sensitive` |
| `DELETE` | `/api/v1/registry-credentials/{id}` | `write:sensitive` |
| `POST` | `/api/v1/registry-credentials/{id}/test` | `write:sensitive` |
| `GET` | `/api/v1/registry-credentials/{id}/repositories` | `read:sensitive` |
| `GET` | `/api/v1/registry-credentials/{id}/tags?repository=...` | `read:sensitive` |
| `GET` | `/api/v1/settings/registry` | `read` |
| `PUT` | `/api/v1/settings/registry` | `root` |
| `DELETE` | `/api/v1/settings/registry` | `root` |
| `GET` | `/api/v1/registry/repositories` | `read` |
| `GET` | `/api/v1/registry/tags?repository=...` | `read` |
| `GET` | `/api/v1/backups?limit=&before=` | `read` |
| `POST` | `/api/v1/databases/{name}/backups` | `write:sensitive` |
| `GET` | `/api/v1/databases/{name}/backups?limit=&before=` | `read` |
| `DELETE` | `/api/v1/databases/{name}/backups/{historyId}` | `write:sensitive` |
| `PUT` | `/api/v1/databases/{name}/backup-schedule` | `write:sensitive` |
| `DELETE` | `/api/v1/databases/{name}/backup-schedule` | `write:sensitive` |
| `GET` | `/api/v1/backups/health` | `read` |
| `GET` | `/api/v1/backups/coverage` | `read` |
| `POST` | `/api/v1/backups/protect-all` | `write:sensitive` |
| `POST` | `/api/v1/backups/encryption-key/ack` | `write:sensitive` |
| `GET` | `/api/v1/storage/orphans` | `read` |
| `DELETE` | `/api/v1/storage/orphans/{id}` | `write:sensitive` |
| `GET` | `/api/v1/backups/drills?service=&volume=&database=&limit=` | `read` |
| `GET` | `/api/v1/backups/drills/{id}` | `read` |
| `POST` | `/api/v1/backups/drills` | `write:sensitive` |
| `POST` | `/api/v1/backups/protection/refresh` | `write:sensitive` |
| `GET` | `/api/v1/apps/{name}/volumes/{volume}/backup-policy` | `read` |
| `PUT` | `/api/v1/apps/{name}/volumes/{volume}/backup-policy` | `write:sensitive` |
| `POST` | `/api/v1/apps/{name}/volumes/{volume}/restore-to` | `write:sensitive` |
| `POST` | `/api/v1/apps/{name}/volumes/{volume}/backups` | `write:sensitive` |
| `GET` | `/api/v1/apps/{name}/volumes/{volume}/backups?limit=&before=` | `read` |
| `DELETE` | `/api/v1/apps/{name}/volumes/{volume}/backups/{historyId}` | `write:sensitive` |
| `GET` | `/api/v1/apps/{name}/volumes/{volume}/backup-schedule` | `read` |
| `PUT` | `/api/v1/apps/{name}/volumes/{volume}/backup-schedule` | `write:sensitive` |
| `DELETE` | `/api/v1/apps/{name}/volumes/{volume}/backup-schedule` | `write:sensitive` |
## CLI
```bash theme={"dark"}
levelrail-cli backup-targets list
levelrail-cli backup-targets get
levelrail-cli backup-targets delete
levelrail-cli backup-targets test
levelrail-cli backup-targets create --name NAME --provider PROVIDER --bucket BUCKET --access-key-id ID --secret-access-key KEY [--endpoint URL] [--region REGION]
# --endpoint is required for r2 and custom, --region is also required for custom.
levelrail-cli backup-targets update --name NAME --provider PROVIDER --bucket BUCKET [--access-key-id ID --secret-access-key KEY] [--endpoint URL] [--region REGION]
levelrail-cli registry-credentials list
levelrail-cli registry-credentials get
levelrail-cli registry-credentials delete
levelrail-cli registry-credentials test
levelrail-cli registry-credentials create --name NAME --registry-host HOST --username USER --password PASS [--expires-at RFC3339]
levelrail-cli registry-credentials update --name NAME --registry-host HOST --username USER [--password PASS] [--expires-at RFC3339]
levelrail-cli registry-credentials repositories
levelrail-cli registry-credentials tags
levelrail-cli registry status
levelrail-cli registry enable --host HOST
levelrail-cli registry disable
levelrail-cli registry repositories
levelrail-cli registry tags --repository NAME
levelrail-cli backups schedule set --target ID --cron EXPR [--retain N] [--retain-days N]
levelrail-cli backups schedule clear
levelrail-cli backups delete
levelrail-cli backups list-all [--limit N] [--before RFC3339]
levelrail-cli backups health
levelrail-cli backups protection [--refresh]
levelrail-cli backups drill run --backup ID
levelrail-cli backups drill list [--app A --volume V | --database D]
levelrail-cli backups drill show
levelrail-cli backups volumes list
levelrail-cli backups volumes policy get|set [--retain-daily N --retain-weekly N --retain-monthly N --pre-hook CMD --post-hook CMD --pause]
levelrail-cli backups volumes restore --backup ID [--target-app APP] [--node NODE] [--new-volume NAME]
levelrail-cli app-volume-backups schedule set --target ID --cron EXPR [--retain N] [--retain-days N]
levelrail-cli app-volume-backups schedule clear
levelrail-cli app-volume-backups delete
```
## Troubleshooting
### Scheduled backup not running or missing
Likely cause: The backup schedule is not set, the backup target is not reachable, or the reconciler has not processed the backup task yet.
How to confirm: Check if a schedule is configured: `levelrail-cli backups schedule ` or check the database's Backups tab in the dashboard. If a schedule exists, check the target: `levelrail-cli backup-targets test ` should report success. For recent backups, check the list: `levelrail-cli backups list --limit 10`.
Fix: Set a backup schedule if one does not exist: `levelrail-cli backups schedule set --target --cron "0 3 * * *" --retain 7`. If the target fails the test, fix the credentials or connectivity to the backup storage and try again. Wait a few minutes for the next reconcile pass to trigger the next backup.
Still stuck: See [Backup targets](/backups-and-storage#backup-targets) for target setup, or check control plane logs for backup errors with `levelrail logs --since 1h | grep backup`.
### Backup target test fails with "connection refused" or "access denied"
Likely cause: The network path to the backup storage is blocked, or the credentials (access key, secret, or bucket name) are incorrect.
How to confirm: From the node running the control plane, test the connection manually. For S3: `aws s3 ls s3:// --endpoint --region ` with the stored credentials. Check the exact error message: `levelrail-cli backup-targets test ` shows the failure reason.
Fix: Verify the credentials match your storage provider (check in your provider's console). Verify the bucket name is correct and exists. If using a custom endpoint, ensure the URL is reachable from the control plane network. Update the target if needed: `levelrail-cli backup-targets update --access-key-id --secret-access-key `, then test again.
Still stuck: See [Backup targets](/backups-and-storage#backup-targets) for each provider's setup steps.
### Backup is old, "Health: 1 missing backup" warning
Likely cause: A scheduled backup failed or is delayed. The most recent backup is older than expected based on the schedule.
How to confirm: Check the backup history: `levelrail-cli backups list --limit 5` and note the timestamp on the most recent backup. Compare it to the cron schedule set on the database. Check for backup health: `levelrail-cli backups health` shows overall status. If health is not OK, check the Backups page in the dashboard for individual backup statuses and errors.
Fix: Wait for the next scheduled backup to run (the reconciler checks every 15 seconds). If the backup is still not running after the scheduled time passes, trigger a manual backup: `levelrail-cli backups trigger --target `. If the trigger fails, check the backup target connectivity as described above.
Still stuck: Check control plane logs for the backup reconciler: `levelrail logs --since 1h backup | tail -30`.
### Restore fails or restored data is wrong
Likely cause: The backup file is corrupted, the restore target has incompatible schema, or the backup verification drill never ran.
How to confirm: List available backups: `levelrail-cli backups list --limit 5`. Check the backup status: each backup should show "completed". Run a restore drill to verify: `levelrail-cli backups drill run --backup `. A drill that fails means the backup may not restore cleanly.
Fix: Use a different backup from the list (backups older than the failed one are usually safer). Run a drill first to catch issues before attempting a real restore. For a real restore: `levelrail-cli backups restore --backup --confirm `. This creates a new database instance from the backup, and you can switch traffic after verifying.
Still stuck: See [Point-in-time restore](/database-upgrades) and [Restore drills](/backups-and-storage) for detailed recovery procedures.
### Registry credentials test fails with "authentication failed"
Likely cause: The username, password, or registry host does not match what the provider has on file, or the token has expired.
How to confirm: Check the credentials are correct: `levelrail-cli registry-credentials get ` shows the registry host and username (password is never returned). Test manually from the control plane: `docker login --username --password ` should succeed.
Fix: Update the credentials: `levelrail-cli registry-credentials update --username --password `. If the token has an expiry, rotate it at your registry provider first, then update here. Test again: `levelrail-cli registry-credentials test `.
Still stuck: See [Registry credentials](/backups-and-storage#registry-credentials) for setup per provider.
## Next steps
Restore, point-in-time recovery, and backup verification.
Connect Backblaze B2, Wasabi, MinIO, and other S3-compatible buckets.
Alert rules, including `backup_missing`.
Deploy apps that use registry credentials.
# Run behind an existing proxy
Source: https://docs.levelrail.com/behind-an-existing-proxy
Run Levelrail behind the proxy that already owns ports 80 and 443, with managed routes for Traefik (including Coolify) and Nginx.
On a server that already runs Coolify, Dokploy or your own nginx, something else
owns ports 80 and 443. Levelrail doesn't fight it. It detects what holds the ports
and generates the configuration to put the dashboard behind it.
## One step setup (Traefik, including Coolify)
When the proxy is Traefik with a file provider directory, Levelrail can manage
the routes itself: one file per domain in the directory Traefik watches,
rewritten on every reconcile pass and removed when the domain goes away.
```bash theme={"dark"}
levelrail-cli proxy setup # dry run: what was detected and what would change
levelrail-cli proxy setup --confirm # apply
levelrail-cli proxy status # per-domain state, certificate, checklist
```
Traefik loads a written file a moment after it appears. `setup --confirm` and
`proxy verify` therefore wait (up to `APP_PROXY_VERIFY_TIMEOUT`, default `8s`)
for the router to show up, and a route that is still not loaded is reported as
the pending step "waiting for the proxy to load the route", not as an error.
Run `proxy verify` again once Traefik has reloaded.
Setup, in order:
1. Detect. It reads the container publishing host port 80 or 443 through the
Docker API (no `docker` CLI): its arguments, mounts, published ports and
extra hosts. A mounted `traefik.yml` is read instead of the arguments when
Traefik loads one. Nothing is defaulted: anything it cannot determine is
listed and setup stops with `409` and the exact gap.
2. TLS upstream. It turns on `tls_terminated_upstream` and sets
`public_https_port` to the host port the proxy publishes for HTTPS. This
instance stops requesting certificates and stops redirecting HTTP itself.
3. Routes. It saves the detected values and writes the route files.
4. Verify. For every route it runs a TLS handshake to this server's public
address with the domain as SNI (issuer and expiry of the certificate the
proxy serves), then a request with the domain as `Host`. When Traefik's API
is published without authentication, it also asks Traefik whether the
router is loaded.
Running setup again changes nothing. Turn it off with `levelrail-cli proxy
disable`, which removes the files it wrote and keeps everything else.
### What is detected on Coolify
Coolify starts `coolify-proxy` with:
```text theme={"dark"}
--entrypoints.http.address=:80
--entrypoints.https.address=:443
--providers.file.directory=/traefik/dynamic/
--certificatesresolvers.letsencrypt.acme.httpchallenge=true
extra_hosts: host.docker.internal:host-gateway
volume: /data/coolify/proxy -> /traefik
```
which gives:
| Setting | Value | From |
| - | - | - |
| `dynamic_dir` | `/data/coolify/proxy/dynamic` | the file provider directory mapped through the mount |
| `entrypoint_http` / `entrypoint_https` | `http` / `https` | the entrypoints published on host ports 80 and 443 |
| `cert_resolver` | `letsencrypt` | the only certificate resolver |
| `upstream_host` | `host.docker.internal` | the extra host; its target (Docker's host gateway) is resolved to validate the listeners |
Each domain gets `/data/coolify/proxy/dynamic/levelrail-managed-.yaml`:
an HTTPS router with the certificate resolver, an HTTP router that redirects to
HTTPS, and a service pointing at the ingress HTTP port (app domains) or the
dashboard listener (the primary domain). Coolify lists these files next to its
own `coolify.yaml` and `default_redirect_503.yaml`; leave them to Levelrail.
### When the listener is on loopback
The proxy runs in a container, so it can't reach a listener bound to
`127.0.0.1`. Setup then stops with `upstream_unreachable` and the exact
drop-in, for example:
```ini theme={"dark"}
# /etc/systemd/system/levelrail.service.d/proxy-upstream-ingress.conf
[Service]
Environment=APP_INGRESS_HTTP_ADDR=172.17.0.1:8088
```
then `systemctl daemon-reload && systemctl restart levelrail`. That address is
internal to the host, not reachable from the internet.
### Safety model
The control plane writes into a directory another product owns, so it is
strict about what it touches:
* Only files named `levelrail-managed-*.yaml` whose first line is
`# managed by levelrail`. A file with the prefix but without the header is
reported and never overwritten or deleted.
* Symlinks are never followed; the directory is resolved once and re-checked
before every change, and a directory that resolves outside its parent is
refused.
* Writes go to a temporary file in the same directory, then `rename`, mode
`0644`, so Traefik never reads a half-written file.
* Domains are validated strictly: no wildcards, ports, paths or separators,
and a domain that would need rewriting to become a file name is rejected.
* If a route cannot be rendered (for example the upstream became unreachable),
the existing file is kept rather than deleted.
* Every write and removal is audited as `proxy_route.written` and
`proxy_route.removed`.
* If the control plane cannot write the directory (another user, a read-only
mount or a systemd sandbox), setup reports `not_writable` with the uid, the
owner and mode of the directory, or the `ReadWritePaths=` line to add.
### Settings
`PUT /api/v1/settings/proxy-integration` sets them by hand, for a Traefik whose
directory is not on a host mount detection can see, or with a TOML static file:
```json theme={"dark"}
{ "integration": "traefik_file", "dynamic_dir": "/srv/traefik/dynamic",
"entrypoint_http": "web", "entrypoint_https": "websecure",
"cert_resolver": "le", "upstream_host": "172.18.0.1" }
```
`levelrail-cli proxy setup --dynamic-dir DIR` covers the common case of a
directory detection could not map.
## From the dashboard
Open Domains, then "Put the dashboard behind your existing proxy". Enter the domain,
check the detected proxy, and copy the configuration. "Check it works"
resolves the domain, confirms it points at this server, and requests
`/healthz` over HTTPS.
## From the CLI
```bash theme={"dark"}
levelrail-cli proxy # who holds 80 and 443
levelrail-cli proxy --domain console.example.com # configuration for that domain
levelrail-cli proxy --domain console.example.com --verify
levelrail-cli proxy --app web # snippet for an app's domain (upstream: the ingress HTTP port)
```
## What it generates
| Proxy | You get |
| - | - |
| Traefik | a dynamic configuration file (Coolify's is `/data/coolify/proxy/dynamic/`) with HTTPS and a Let's Encrypt router |
| nginx | a server block with WebSocket headers, plus the certbot command |
| Caddy | a Caddyfile site block |
If the proxy runs in a container and the dashboard listens on loopback, the
guide also gives a systemd drop-in that moves the dashboard to that network's
gateway address. That address is internal to the host, not reachable from the
internet, and a drop-in survives upgrades.
## Certificates once a proxy owns TLS
When TLS is terminated by the proxy, the platform reports one certificate state
for each domain: the certificate the proxy actually serves (issuer, expiry,
"renews automatically"), read from the last route verification. Nothing that
this server stored earlier is shown or alerted on:
* A stale certificate request failure from an earlier attempt, the "another
service answers port 80" guidance and the DNS-01 provider prompt are hidden,
and the recorded failure is cleared when you enable proxy mode.
* The internal certificate this server stored before the proxy took over is not
listed under **Settings > Certificates**, in the security checks or in the
domain doctor, and `cert_expiring`, `cert_expiry` and `cert_renewal_stalled`
alert rules ignore it.
* `GET /api/v1/domains` and `GET /api/v1/certificates` carry a `tls_owner`
field (`acme`, `internal`, `proxy`, `custom`); `levelrail-cli domains
certificates` prints it as the OWNER column.
If the proxy certificate has not been seen yet, run **Verify** on the proxy
page (or `levelrail-cli proxy --verify`).
## Traffic controls behind a proxy
Maintenance mode, basic auth, the auth gate, the WAF and rate limit, redirects
and custom error pages are all applied by the built-in ingress. Behind a proxy
they work only when the proxy really hands the domain's requests to that
ingress. A route file being written is not enough: another router on the same
proxy (for example a container label left over from Coolify) can win the match
and send visitors straight to an old container.
Every domain in `GET /api/v1/domains` (and every entry of `enforcement` in
`GET /api/v1/apps/{name}/environment-domains`) carries an `enforcement` block:
```json theme={"dark"}
"enforcement": {
"mode": "proxy:traefik",
"maintenance": false, "basic_auth": false, "auth_gate": false,
"waf": false, "redirect": false, "error_pages": false,
"reason": "route_unverified"
}
```
| Mode | When | Maintenance | Basic auth | Auth gate | WAF and rate limit | Redirect | Error pages |
| - | - | - | - | - | - | - | - |
| `levelrail` | The built-in ingress serves the domain directly | yes | yes | yes | yes | yes | yes |
| `proxy:traefik` | A managed route file exists and the last **Verify** proved the requests reach the built-in ingress | yes | yes | yes | yes | yes | yes |
| `proxy:traefik` | A managed route file exists but delivery is not proven (`reason: route_unverified`) | no | no | no | no | no | no |
| `proxy:external` | TLS is terminated by a proxy this server does not manage (nginx, Caddy, a hand written Traefik file) (`reason: proxy_unmanaged`) | no | no | no | no | no | no |
| `none` | The domain is not routed yet (`reason: not_routed`) | no | no | no | no | no | no |
"no" means the platform cannot confirm it, so the dashboard shows the saved
setting as **Saved, not enforced** with the reason, never as active. The
setting itself is kept and starts working as soon as delivery is proven.
To get a managed Traefik route to `yes`, open the proxy page and run
**Verify** (or `levelrail-cli proxy --verify`). Verification sends a request
through the proxy that only the built-in ingress answers; when it comes back,
the route is proven. Managed routers are written with a high priority so they
win over label routers for the same host. If verification still fails, look
for another router on the proxy that claims the host with an even higher
priority.
For nginx or Caddy in front, point the site at the ingress HTTP listener as
shown in the generated snippet. The controls then apply, but the platform
cannot verify it and keeps reporting them as not enforced.
## Links and the public HTTPS port
When a proxy owns 80 and 443 and forwards to this ingress on its own ports
(for example `8088` and `8443`), the certificate lives on the proxy and clients
reach the proxy on 443. Set **Public HTTPS port** under **Domains > Platform
ingress** (or `levelrail-cli settings ingress set --public-https-port 443`) so
every link the platform builds (automatic URLs, preview URLs, emailed links)
uses the port clients actually use. `443` produces links with no port suffix,
any other value is appended, and `0` keeps the ingress listen port.
## Several instances behind one proxy
Each instance can't own 80 or 443 and must not run ACME. Give every instance
its own ingress ports, let one proxy hold 80 and 443, and turn on **TLS is
terminated by a proxy in front** (`--tls-terminated-upstream`) on each:
```bash theme={"dark"}
levelrail-cli settings ingress set --tls-terminated-upstream --public-https-port 443
```
In this mode the instance never requests certificates (a stored ACME setting is
kept but skipped), builds `https` links with the public port (443 when unset),
and does not redirect HTTP itself because the proxy does. Set
`APP_INGRESS_TRUSTED_PROXIES` to the proxy's address range so `X-Forwarded-Proto`
and `X-Forwarded-Host` are honoured only from it, never from the open internet.
The dashboard shows "HTTPS is handled by your proxy" instead of a certificate
error. DNS for each domain still points at the proxy host.
Traefik, one host rule per domain and one backend port per instance:
```yaml theme={"dark"}
http:
routers:
team-a:
rule: Host(`a.example.com`) || HostRegexp(`^.+\.a\.example\.com$`)
entryPoints: [websecure]
tls: { certResolver: letsencrypt }
service: team-a
team-b:
rule: Host(`b.example.com`)
entryPoints: [websecure]
tls: { certResolver: letsencrypt }
service: team-b
services:
team-a:
loadBalancer:
servers: [{ url: "http://127.0.0.1:8088" }]
team-b:
loadBalancer:
servers: [{ url: "http://127.0.0.1:8089" }]
```
Instance A runs with `APP_INGRESS_HTTP_ADDR=:8088`, instance B with
`APP_INGRESS_HTTP_ADDR=:8089`. Forward to the HTTP port: TLS ends at the proxy.
## Troubleshooting
### Domains served by the proxy are not routing correctly
Requests to the domain arrive at the proxy but do not reach the right Levelrail app.
Likely cause: the proxy's route rule does not match the Host header, or Levelrail is not listening on the port the proxy is forwarding to.
How to confirm: check the proxy configuration (Traefik, nginx, Caddy) and verify it has a route for the domain pointing to the correct Levelrail port (e.g., 8088). Also check that Levelrail is actually listening on that port with `sudo ss -tlnp | grep 8088`.
Fix: verify Levelrail is running and listening on the correct port (set with `APP_INGRESS_HTTP_ADDR` or via the installer). Then add or update the proxy's route to forward the domain to that port. Refer to [Example proxy configs](/behind-an-existing-proxy#example-proxy-configs) above for the correct syntax for your proxy.
### The dashboard shows mixed HTTP/HTTPS content
The browser blocks some resources because they are loaded over HTTP while the page came over HTTPS.
Likely cause: APP\_INGRESS\_TRUSTED\_PROXIES is not set, so Levelrail does not trust the `X-Forwarded-Proto` header from the proxy and thinks requests are coming over HTTP.
How to confirm: open the browser's developer console (F12) and check the Network tab for mixed content warnings.
Fix: on the control plane, set `APP_INGRESS_TRUSTED_PROXIES` to the IP range or address of your proxy. For example, `APP_INGRESS_TRUSTED_PROXIES=10.0.0.0/8` for a private network, or the proxy's specific IP if it is static. After setting it, restart Levelrail: `sudo systemctl restart levelrail`. Once Levelrail trusts the proxy's `X-Forwarded-Proto` header, it will serve links over HTTPS.
### The dashboard is unreachable from outside but works on the server
The proxy is not forwarding the dashboard address to Levelrail, or Levelrail is not listening on the right address.
Likely cause: the proxy's configuration does not include the dashboard domain, or Levelrail's dashboard port (default 8080) is only listening on localhost.
How to confirm: check the proxy's route configuration to see if the dashboard domain is included. Also check Levelrail's systemd unit with `systemctl cat levelrail` and look for `APP_HTTP_ADDR` (should not be `127.0.0.1` when running behind a proxy).
Fix: add a proxy route for the dashboard domain pointing to Levelrail's dashboard port (8080 by default, or the port set with `APP_HTTP_ADDR` or `LEVELRAIL_DASHBOARD_PORT` during install). Set `APP_HTTP_ADDR=127.0.0.1:8080` (local only, since the proxy is the external listener) if it is not already. The installer normally handles this for coexist mode; if you set up the proxy afterward, manually add the route.
Still stuck? Check the [Installing behind an existing proxy](/installing#server-already-runs-a-proxy) section or attach your proxy config to an issue on [GitHub](https://github.com/glincker/levelrail/issues).
## API
`GET /api/v1/system/reverse-proxy?domain=D&proxy=traefik&verify=true`, or
`?app=NAME` for an app's domain.
Managed routes: `GET /api/v1/system/proxy-integration` (detection, settings,
ingress, per-domain state and the setup checklist), `POST .../setup` with
`{"confirm": true}`, `POST .../apply`, `POST .../verify?domain=D`, and
`PUT /api/v1/settings/proxy-integration`. Reading needs the read ability,
every write needs root. `APP_PROXY_VERIFY_TIMEOUT` bounds each probe
(default `8s`).
The port and mode above are `public_https_port` and `tls_terminated_upstream` on
`PUT /api/v1/settings/ingress`.
The check only probes a domain that resolves to this server's own public
address.
# Brand kit
Source: https://docs.levelrail.com/brand
Logos, colors and ready-made social images for Levelrail, free to use when you write about it, share it or link to it.
Everything here is free to use when you write about Levelrail, post it, or link to it. Right-click an image to save it, or use the file links.
## Logo
The mark is three stacked rails. The wordmark is the mark plus the name in capitals.
| Mark | Mark on dark | Wordmark (for dark) | Wordmark (for light) |
| - | - | - | - |
| ![Levelrail mark]()
| ![Levelrail mark on a dark square]()
| ![Levelrail wordmark]()
| ![Levelrail wordmark]()
|
Files: [mark.svg](/assets/brand/levelrail-default.svg), [mark-1024.png](/assets/brand/png/mark-1024.png), [mark-on-dark-1024.png](/assets/brand/png/mark-on-dark-1024.png), [wordmark-for-dark.png](/assets/brand/png/wordmark-for-dark.png), [wordmark-for-light.png](/assets/brand/png/wordmark-for-light.png).
Keep clear space around the mark of at least a quarter of its width. Do not recolor it, stretch it, or add effects.
## Colors
| Name | Hex | Use |
| - | - | - |
| Deep | `#06232C` | Shadows, dark surfaces |
| Rail 1 | `#084F67` | Bottom rail, backgrounds |
| Rail 2 | `#107292` | Primary accent on light, middle rail |
| Rail 3 | `#58B1CE` | Highlights, top rail |
| Bright | `#2FB3DC` | Primary accent on dark |
## Social images
Link preview cards are 1200x630, the size most platforms expect. The platform banners below are rendered at 2x so they stay sharp on high density screens. The sizes listed are the logical size.
### Headlines
Pick the line that fits where you are posting.
| Image | Where |
| - | - |
| [Your own Vercel. One binary.](/assets/og-home.png) 1200x630 | Link previews, the default |
| [Push to git. Get a running app.](/assets/brand/social/headline-push-1200x630.jpg) 1200x630 | Link previews, the plain version |
| [One command. Your own cloud.](/assets/og/install.jpg) 1200x630 | Install guides |
| [Leaving Coolify? Bring your apps.](/assets/og/migrate.jpg) 1200x630 | Migration posts |
| [311 templates. One click each.](/assets/og/templates.jpg) 1200x630 | Template posts |
| [Canary deploys, no service mesh.](/assets/og/canary.jpg) 1200x630 | Deploy strategy posts |
| [Idle apps sleep. Requests wake.](/assets/og/sleep.jpg) 1200x630 | Resource and cost posts |
| [Ask your server what broke.](/assets/og/mcp.jpg) 1200x630 | AI and MCP posts |
![Levelrail social card: Your own Vercel. One binary.]()
### Sizes per platform
| Platform | Image |
| - | - |
| GitHub social preview | [1280x640](/assets/brand/social/github-social-1280x640.jpg) |
| X card | [1200x675](/assets/brand/social/twitter-card-1200x675.jpg) |
| X header | [1500x500](/assets/brand/social/x-header-1500x500.jpg) |
| LinkedIn banner | [1584x396](/assets/brand/social/linkedin-banner-1584x396.jpg) |
| Product Hunt gallery | [1270x760](/assets/brand/social/producthunt-gallery-1270x760.jpg) |
| dev.to cover | [1000x420](/assets/brand/social/devto-cover-1000x420.jpg) |
| Square (Instagram, Discord) | [1080x1080](/assets/brand/social/square-1080x1080.jpg) |
## One line to use
> Levelrail is a self-hosted deploy platform in a single Go binary: push to git, get HTTPS, live logs, metrics and one-click rollback.
Licensed under Apache 2.0. Source: [github.com/glincker/levelrail](https://github.com/glincker/levelrail).
# Build node routing
Source: https://docs.levelrail.com/build-node-routing
Which node a deploy's build runs on, how to mark a node build-only, and what to do when a single node is under sustained load.
On a single node, every build runs locally, alongside the control plane and every running app. Nothing on this page needs deciding until you add a second node.
## The `accepts_build_workloads` flag
Every node has an `accepts_build_workloads` flag. It is a routing preference, not a placement rule and not a guarantee. Set it from the node's detail page in the dashboard, with the CLI, or with `PUT /api/v1/nodes/{id}/workloads`:
```bash theme={"dark"}
levelrail-cli nodes workloads --accepts-app=BOOL --accepts-build=BOOL
```
Both flags are required on every call, because the command replaces both values. A new node accepts app workloads but not build workloads until you enable it.
What the flag does not do:
* It does not move apps or databases already running on the node.
* It does not stop a build from running on a node without it. If no node that accepts builds is online, the build runs on the primary node instead of failing, so a build node going offline never fails a deploy.
## How a build picks its node
For each build, the control plane looks at every registered node with `accepts_build_workloads` set:
1. If none has it set, the build runs locally, as on a fresh single-node install.
2. If at least one is online, the online one with the lexicographically smallest ID runs the build. This is a fixed tie-break, not load-based scheduling. Levelrail does not do bin-packing, affinity or autoscaling.
3. If some have it set but none is online, the build falls back to the primary node.
When a build runs on another node, the built image is streamed back and loaded into the control plane's own image store, so the rest of the deploy works the same as for a local build.
## Mark a node build-only
A build on a node that already serves traffic competes with that traffic for CPU and disk for as long as the build runs. To keep builds off your production containers:
```bash theme={"dark"}
levelrail-cli nodes workloads --accepts-app=false --accepts-build=true
```
Use `--accepts-app=true` instead if the node should also run apps.
```bash theme={"dark"}
levelrail-cli nodes workloads --accepts-app=true --accepts-build=false
```
A node provisioned with `nodes provision --role build` or `nodes ssh-provision --role build` is set up as a build node when it enrolls. See [Node provisioning](/node-provisioning).
## The Nodes page suggestion
On a single-node install, the Nodes page can suggest adding a second node when that node's CPU or disk use has been high for a sustained period. It reuses the existing node resource usage alert (`APP_ALERT_NODE_CPU_THRESHOLD_PERCENT`) and the disk space alert, with no separate threshold. It appears only when telemetry is configured and only while there is exactly one node.
## See also
* [Multi-node](/multi-node): enrolling nodes and managing placement.
* [Observability](/observability): the node alerts behind the suggestion.
To cap how long builds run and how many run at once on any node, see [Build performance](/build-performance).
# Build performance
Source: https://docs.levelrail.com/build-performance
Cap build time and parallel builds, and read the per-app build duration trend and cache hit rate.
Every build is timed, counted against a concurrency limit, and recorded. This page covers the two limits, what is recorded per build, and where to read the trend.
## Timeout and concurrency limit
| Limit | Environment variable | Default |
| - | - | - |
| Build timeout | `APP_BUILD_TIMEOUT` | `30m` |
| Concurrent builds | `APP_BUILD_MAX_CONCURRENT` | `2` |
`0` turns a limit off. A build past the concurrency limit waits in a queue, first in first out, and the wait is recorded but does not count against its timeout. A build that runs past the timeout is stopped and fails with the `build_timeout` code (see [Deploy failures](/deploy-failures)).
To change a limit without restarting the control plane, use **Settings, Build performance** in the dashboard (root accounts only) or the CLI. A saved value wins over the environment variable until you reset it.
```bash theme={"dark"}
levelrail-cli apps build-limits show
levelrail-cli apps build-limits set --timeout 45m --max-concurrent 4
levelrail-cli apps build-limits reset
```
The limits apply to builds that start after the change. Builds already running keep the timeout they started with.
## What is recorded
Each finished build stores its app, start time, duration, queue wait, outcome (`success`, `failed`, `timeout` or `canceled`), and how many of its build steps were served from cache. The newest 200 builds per app are kept. A build canceled while it is still queued never ran and is not recorded.
Builds are keyed by the image repository name, which is the app name for git and manual deploys.
## Reading the trend
The app's **Deploys** page shows a build performance card for the last 20 builds: median (p50), p95, cache hit rate and the number of failed or timed out builds, with a bar per build.
* Percentiles use successful builds only. Failed, timed out and canceled builds are counted, not timed.
* Cache hit rate is cached steps over all steps across successful builds. It shows `n/a` when no build reported steps.
* A successful build with a negative duration (clock skew) is counted but left out of the timings.
From the CLI:
```bash theme={"dark"}
levelrail-cli apps build-stats web --limit 50
```
Over the API, `GET /api/v1/apps/{name}/build-stats?limit=50` returns the same trend plus every run, and `GET /api/v1/build-limits` returns the limits in effect. See the [API reference](/api-reference).
Remote registry cache hits are reported by BuildKit like local ones, so [build cache](/object-storage) setup shows up here as a rising cache hit rate.
# Canary deploys
Source: https://docs.levelrail.com/canary-deploys
Send a share of an app's traffic to a new image before promoting it, with one-command promote and abort.
Run a new image next to the current release and send it a percentage of
requests. If it looks healthy, promote it. If not, abort and every request
goes back to the release that was already serving. Nothing changes for the
stable release until you promote.
## How it works
* Starting a canary creates a hidden clone of the app (`--canary`) with
the same environment variables and secrets, running the new image. It does
not appear in the apps list and has no domains of its own.
* Ingress splits each request between the app and the canary by weight (1 to
99 percent). The split is per request, not per user, so one visitor can see
both releases. The split follows a repeating cycle of 100 requests, so a
weight of 20 sends exactly 20 of every 100 requests to the canary. A handful
of requests may all land on one release.
* A canary that is not running or not ready receives no traffic, so a bad
image cannot take the app down. Weight 0 pauses the canary without removing it.
* **Promote** deploys the canary image to the app through the normal deploy
path (readiness gate, rollback target, freeze and protection checks), then
removes the clone.
* **Abort**, or deleting the app, removes the clone.
## Limits
* One canary per app.
* Image apps only, with a single replica and no pinned host port. Apps built
from source, stopped apps, and apps under a deploy freeze are refused.
* The canary has its own container and its own data connections. If the app
writes to a database, the canary writes to it too.
## CLI
```
levelrail-cli apps canary start web --image myorg/web:next --weight 10
levelrail-cli apps canary weight web --weight 50
levelrail-cli apps canary status web
levelrail-cli apps canary promote web
levelrail-cli apps canary abort web
```
## Dashboard
App, Deploys: the Canary release card has the image and traffic share form,
and Promote and Abort buttons while a canary is running.
## API
`GET`, `POST`, `PUT` and `DELETE /api/v1/apps/{name}/canary`, and
`POST /api/v1/apps/{name}/canary/promote`. All need the `deploy` ability, and
`GET` needs `read`. See the [API reference](/api-reference).
## Verified
Run on a real control plane with Docker and embedded Caddy: weights of 30 and
20 produced exactly 30 and 20 canary responses per 100 requests, weight 0
sent everything to the stable release, and promote left only the new image
serving. A weight change reaches ingress within seconds. A new canary takes up
to about 30 seconds to start its container.
## Troubleshooting
### Cannot start a canary on this app
**Likely cause:** The app is built from source, has multiple replicas, a pinned host port, is stopped, or is under a deploy freeze.
**How to confirm:** Run `levelrail-cli apps canary start --image `. The error will state which limitation applies.
**Fix:** Canaries work only with prebuilt image apps. If the app is built from source, deploy the new image manually first (push to a registry, then deploy by tag). If it has multiple replicas, scale it to one. If it has a pinned host port, remove it first.
### Canary is not receiving traffic
**Likely cause:** The canary container is not running or failed its readiness check, or the weight is set to 0.
**How to confirm:** Run `levelrail-cli apps canary status ` and check the canary app's container status in the dashboard (it is named `--canary` but not listed in the apps view).
**Fix:** Check the canary app's logs with `levelrail-cli apps logs --canary -f` for startup errors. If the image is bad, abort the canary and start a new one with a different image. If the weight is 0, increase it with `levelrail-cli apps canary weight --weight 10`.
### Promote gets stuck or times out
**Likely cause:** The canary image does not pass the readiness gate (same as a normal deploy), or there is a deploy freeze blocking it.
**How to confirm:** Run `levelrail-cli apps deploys show ` and check the failure details. Look for a deploy freeze in Settings > Deploy safety.
**Fix:** Check the canary image logs and fix any startup issues. If there is a deploy freeze, promote after the freeze window ends, or override it with a reason.
### Still stuck
For help with a stuck canary deploy, check [Deploy failures](deploy-failures.md) for the failure code, or run `levelrail-cli apps diagnose ` and share its output.
# CapRover alternative: self-hosted deploys without Docker Swarm
Source: https://docs.levelrail.com/caprover-alternative
Levelrail is an open source CapRover alternative: plain Docker instead of Swarm, an agent per node, built-in metrics and logs, one-click rollback.
**Keep the simple CapRover workflow, drop the Swarm dependency**
Levelrail runs apps on plain Docker containers with its own reconciler and a small agent per extra server. Git push to deploy, automatic HTTPS, rollback and an importer for your CapRover apps.
[Get started](/getting-started) | [View on GitHub](https://github.com/glincker/levelrail)
```bash theme={"dark"}
curl -fsSL https://levelrail.com/install.sh | sudo sh
```
## Where Levelrail differs
Apps run as ordinary containers. A level-triggered reconciler compares desired and observed state and converges, recording a reason after every pass.
Extra servers run a small agent that dials out to the control plane with mutual TLS, so managed servers need no inbound port.
Node-local metrics, full-text log search, crashloop detection and alerts to eighteen channels ship in the core, with deploy markers on the charts.
## Bring your apps with you
The importer reads your existing instance through its own HTTP API and never changes anything there. Run it with --dry-run first.
```bash title="install, import, verify" theme={"dark"}
curl -fsSL https://levelrail.com/install.sh | sudo sh
# control plane running as a systemd service
levelrail-cli import platform caprover --url https://captain.example.com --dry-run
# dry run: report only, nothing is written here or changed there
levelrail-cli import platform caprover --url https://captain.example.com --only web,api
# apps and databases created, ready to deploy
```
## CapRover and Levelrail, side by side
CapRover is a long-standing project with a large install base and a one-click app store. These are the design differences.
| | CapRover | Levelrail |
| - | - | - |
| Server management | Docker Swarm | Agent per node over gRPC with mutual TLS, Docker Engine API |
| Orchestration | Swarm services | Level-triggered reconciler over plain Docker |
| Ingress | nginx | Caddy embedded in the control plane |
| Runs as | A web application running as a Swarm service | One Go binary with embedded SQLite |
| License | Open source | Apache 2.0, one edition |
[Read the full comparison](/comparison)
## Move over safely
One script checks the host, installs Docker if missing and starts the control plane.
Point it at your CapRover URL with --dry-run. The importer is read-only against the source.
Import a subset with --only, deploy it, and run both for a while before switching DNS.
## Frequently asked questions
No. It runs plain Docker containers on each node. Placement across servers is explicit, and cordon and drain are supported.
Use levelrail-cli import platform caprover with your CapRover URL. It is read-only against the source and supports --dry-run and --only.
Levelrail ships a curated template catalog and is expanding it, but CapRover's app store is larger today. Check that the services you need are covered.
Yes. Apache 2.0, one edition, nothing behind a paywall.
## Try it beside your CapRover setup
Import read-only, run both, and keep what works.
[Get started](/getting-started) | [View on GitHub](https://github.com/glincker/levelrail)
# Case studies: measured results, not testimonials
Source: https://docs.levelrail.com/case-studies
Real Levelrail measurements: a 500-app idle benchmark and multi-node verification against real Docker daemons, plus what is not proven yet.
**Measured results, stated honestly**
Levelrail is pre-release and has no customer logos yet, so instead of testimonials this page shows what has actually been measured and verified, and what has not.
[Run your own test](/getting-started) | [View on GitHub](https://github.com/glincker/levelrail)
## What has been verified
In the idle benchmark, going from 0 to 500 suspended apps moved resident memory from about 68 MB to about 91 MB and CPU from 0.03% to 1.6%. It ran on an Apple M4 Max with a development build and no containers running, so expect different numbers on a Linux VPS. The method is reproducible.
Enrollment, cordon, drain with real container relocation and explicit placement pinning were run with two real agent processes, each with its own Docker daemon.
## What is not proven yet
Single node is the best tested path. The WireGuard mesh and cross-host remote transport beyond the join flow are still unverified, there is no Linux release-build idle measurement yet, and public ACME has one live run with renewal and wildcards unverified. The feature status page lists the evidence behind each area.
Fewer real-world deployments also means less community knowledge than projects that have existed for years.
## Be the first case study
If you run Levelrail on a real workload, we would like to hear what broke and what worked. Open a discussion or an issue on GitHub, and if you are happy to be named, we will write it up with your numbers.
## At a glance
* **68 MB**: RSS at 0 apps, dev build
* **91 MB**: RSS at 500 suspended apps
* **1.6%**: CPU at 500 suspended apps
* **500**: apps in the idle benchmark
## Frequently asked questions
Because the project is pre-release and has no public production customers yet. This page only shows measurements and verifications that exist.
From a documented benchmark script (`scripts/bench-idle.sh`) you can run yourself. The [performance page](/performance) lists the method, the machine and the caveats.
They were not measured against other platforms here, so this page makes no comparative claim. Measure your own setup with the same method if you want a comparison.
## Add your results
Run it, measure it, and tell us what you find.
[Run your own test](/getting-started) | [View on GitHub](https://github.com/glincker/levelrail)
# Changelog
Source: https://docs.levelrail.com/changelog
Every Levelrail release with highlights, fixes and upgrade notes.
**Bug Fixes**
* add the frontmatter description the docs build requires to design tokens (#1180)
**Bug Fixes**
* restore the all-features test baseline in the CLI and bound two integer conversions (#1177)
* updates channel resolution, Edge dead end and Cloud Run guides for launch (#1175)
**Features**
* 0.6 upgrades, dashboard v2, standalone domains, secrets v2 and reliability fixes (#1164)
* attribute dashboard memory to the control plane and unmanaged containers
* CLI for standalone domains and the server address
* cost showback per app, project, environment and team
* coverage-first backups page, protect-all, and orphaned data reclaim
* dashboard restart, dock and CLI console, forward auth, responsive pass
* dock as fixed overlay widget with float, bottom and side modes
* docked bottom panel hosting the AI chat with ctrl+j toggle
* fold the create-app repo fields into a summary once a repo is picked (#1168)
* Get started hub with live setup steps, server-side progress and CLI
* levelrail-cli tui, a k9s-style terminal UI for apps, deploys, logs and actions
* log explorer with saved queries, error signatures and log alert rules
* mask sensitive env everywhere, step-up reveal, one environment editor
* MCP prompts, resources and --print-config for levelrail-mcp
* mobile pairing API and built-in sign-in provider for the auth gate (#1173)
* network page groups this server as a real node and mesh can be enabled from the UI and CLI
* one click quick actions in the web CLI console
* per-domain auth gate (forward auth) in the embedded ingress
* per-domain auth gate (forward auth) with header spoof stripping
* read-only host diagnostics in the web CLI console
* redesign deploy history rows and the deploy page around the logs (#1165)
* resource advisor dashboard, Optimise panel, beta settings and docs
* resource advisor engine, API, CLI and MCP tool behind a beta flag
* responsive pass for phone and tablet across dashboard, lists and settings
* restart the control plane from the dashboard
* show the live preview image and commit subject on the app overview (#1167)
* sign-in attribution footer with GLINR Studios and TheAuth marks
* sign-in screens and remaining charts use design tokens
* single-node placement shows this server, nodes table and settings index regrouped
* split-screen sign-in with a rotating brand showcase from the TheAuth visuals (#1169)
* standalone domains proxy to a port, address, URL, app or container
* sticky dashboard header, dark elevation tokens and GLINR CSS tokens
* suggestion-driven New app wizard with repository analysis
* system restart CLI command with --wait and status
* triage model, certificate guardrails and resources-first dashboard home
* turn AI chat on from Settings, Beta features, and fix header right alignment
* two-mode add domain dialog, standalone rows and server address card
* upgrade blockers with fixes, upgrade now, scheduled and automatic upgrades
* web CLI console over WebSocket, scoped to the platform CLI
* web CLI console route, dock tab and beta toggle
**Bug Fixes**
* auth gate keeps the client query on the auth subrequest
* deployments rows open the deploy page, and long commit messages no longer flood the drawer (#1163)
* dock AI tab explains when the ai-chat experimental flag is off
* keep placement display fields out of the MCP client types to stay under the tool token budget
* keep pre-library API tokens working after upgrade
* merge edge channel modelling and stale incident fixes with scheduled upgrades
* picking a repo prefills the create-app form and collapses the repo grid (#1166)
* sanitize terminal escape sequences in web console output
* serve /metrics and jwks, restart after swap on self-upgrade, restore-as-new owner, ungate data export CLI
* serve internal fallback certs when ACME fails, validate ACME contact, abort never-listening deploys
* stale data copy incidents, plain-language copy failures, and edge channel update messaging
* tcp readiness fallback, server help flags, backup target validation, version stamp
* wait for proxy route load in verify, warn on alert rules with no channel
**Documentation**
* standalone domains, server address, and confirmation fixes
**Bug Fixes**
* dockerguard builds on Windows, and CI cross-compiles the CLI (#1157)
**Features**
* 0.4 compose, cutover, previews, teams, backups, observability, multi-node, upgrades (#1140)
* 0.5 dashboard performance, smart deploy, proxy-managed domains, shell and database UI, MCP (#1147)
* **api:** publish an OpenAPI 3.1 spec generated from the route table (#1037)
* attention center, device login approval, setup redesign, release rollback (#1077)
* brand kit page, headline variants and per-page social cards (#1094)
* canary deploys with weighted traffic (#1065)
* claim containers from a docker inspect snapshot (no platform token) (#1092)
* CLI insights commands, reconciler half-success tests, 19 catalog templates (#1033)
* coolify migration data path, domain wizard, environment domains, db viewer (#1074)
* database access and network, IAM policy builder, app import wizard (#1091)
* **docs:** animate the product mock, add mock carousel, scroll reveal and stats strip (#1041)
* **docs:** favicons, guides, template gallery, announcements feeds (#1060)
* **docs:** flat full-window hero mock, seam fix and Mastra-style FAQ (#1038)
* **docs:** hero uses the product mock, .well-known ships, light-mode fix (#1035)
* **docs:** homepage hero, feature cards, install chip, nav fixes (#1024)
* **docs:** site overhaul with new landing pages, grouped nav, mock kit, legal and agent-readiness (#1034)
* domains, DNS, proxy coexist, traffic controls and platform hardening (0.3) (#1125)
* extract generic packages into a nested kit module (#1014)
* first-run fixes: hide domains from search engines, proxy coexist guide, IPv4 public host (#1102)
* functions, requests that wake a sleeping app wait and are replayed (#1068)
* glinr-bot policy evaluator, shadow-mode workflow and client-id for app tokens (#1100)
* install cosign by default and add upgrade-note for manual binary swaps (#1110)
* legacy passkeys sign in and OAuth keeps built-in callback URLs under the library engine (#1029)
* library auth engine behind APP\_AUTH\_ENGINE with backfill (#1015)
* list case-duplicate email groups in auth-backfill, add go/no-go checklist (#1023)
* managed postgres pgvector variant (#1078)
* migrate auth to the theauth library behind default-off flags (#1021)
* move upgrade into kit and brand-drive agent provisioning names (#1020)
* perf, exposure audit, migration hub, upgrade history, external databases, attention v2, fleet overview (#1087)
* public HTTPS port and TLS terminated upstream ingress settings (#1117)
* regenerate OG and social images on the petrol brand (#1093)
* roles and guest access, global environments, environment-aware IAM and AI control (#1048)
* scripts/build-linux.sh builds the embedded-web binary with the version stamped (#1112)
* security center, session management and sign-in hardening (#1133)
* setup, HTTPS, token and image auto-update improvements (#1062)
* sign in with a code and new-browser approval (#1115)
* sleep idle apps and wake them on the next request (#1066)
**Bug Fixes**
* \[HIGH] Server-Side Request Forgery (SSRF) in Git provider integrations (#1051)
* bump the mealie template to v3.28.0 (#1019)
* database explorer schema for large databases, mark explorer beta (#1080)
* deflake scheduledtask replace and agent TTY close tests (#1053)
* DNS check shows what each resolver answered, a timestamp and a propagating state (#1123)
* docker snapshot apps stay image apps with a load hint, neutral secret hint (#1096)
* **docs:** code block copy button back in the corner, inline code tint only inline (#1042)
* **docs:** readable light-mode nav panel and a balanced footer (#1040)
* **docs:** use the app metrics screenshot in the home hero (#1026)
* egress sidecar gets NET\_RAW, e2e cleanup removes service volumes (#1059)
* exact 1200x630 social cards with a CTA, og:site\_name, short descriptions, bolder favicon (#1097)
* explain and link the private GitHub App case when adding an organization (#1130)
* explain and offer a fix when real certificates cannot work on non-standard ingress ports (#1114)
* fall back to the embedded default logo and colors when brand.yaml leaves them empty (#1075)
* findings from real-server QA (sign-in, firewall, redeploy, templates, status UI) (#1043)
* git push deploys fall back to the GitHub App installation token (#1145)
* glinr-bot talks to the GitHub API over net/http instead of spawning gh (#1103)
* levelrail.com uses the transparent mark like the control plane; add 96px icon and manifest (#1104)
* library sign-in returns the second-factor step for users with TOTP (#1025)
* match legacy device and recovery code behavior in library auth mode (#1027)
* README typing banner on the petrol brand with the new headline (#1095)
* repair 7 catalog image pins that no longer resolve (#1031)
* serve plain HTTP for a TLS-terminating proxy and show setup on the Proxy page (#1127)
* short delta percentages and proxy-served certificate in app health (#1141)
* unblock CI on main (execgate allowlist for self-upgrade files, skip test-only packages in go build) (#1148)
* verify moved images by content, not image ID (#1121)
**Performance**
* cut PR CI wall time (live suite gate, parallel fleet, plan job) (#1090)
* optimize string concatenation in joinErrors (#1061)
* scope migrations by table, guards lane, green-based nightly, scoped hooks, remote Go cache (#1131)
**Documentation**
* add see-also sections to multi-node and cli-reference pages (#1030)
* full pass for accuracy, components and coverage (#1036)
* polish code blocks, tables, callouts and doc components (#1028)
* README and social card tuned for conversion (#1063)
**Features**
* harden ingress edge, hold pages, and systemd socket activation (#999)
* orphan reaper, retried teardown paths, cascading project/env/org delete (#1008)
**Bug Fixes**
* database data safety, remote WAL, guarded Postgres major upgrade, DR runbook (#1009)
* git private-network opt-in with its own env, unbreak e2e fixtures (#1010)
* harden alerting, auth and day-2 paths (#1007)
* local-node replicas published on loopback are balanced, not dropped as remote (#1003)
* publish prerelease npm builds as latest until a stable release exists (#1002)
**Features**
* one-command npm bootstrap and opt-in hands-off releases (#995)
* route ingress to apps on remote nodes over the WireGuard mesh (#996)
**Bug Fixes**
* app delete tears containers down and retries until gone (#991)
* **auth:** token ownership, OAuth state binding with PKCE, atomic device redeem (#981)
* balance multi-replica apps by default and route the local node's replicas (#994)
* Homebrew formula exec bit, npm trusted publishing, split publish jobs (#992)
* make node enrolment and mesh work out of the box on the agent image (#983)
* security review hardening (CSRF, cert purge churn, placeholder escape, git SSRF, hardening default) (#997)
**Features**
* CLI and agent distribution, Windows CLI, agent skill files, update-check and main CI fixes (#972)
* zero-DNS sslip.io HTTPS, db cert storage, http redirect (#977)
**Bug Fixes**
* data safety hardening for databases, backups, restore and volumes (#980)
* mention levelrail-cli install and login in install.sh summary (#970)
* reject cross-origin cookie-authenticated mutations (#976)
* zero-downtime hardening for pinned ports, rolling, secrets, replicas (#978)
**Documentation**
* add seven tutorials for search and fix restore-as-new race (#979)
* fix MCP tool count and ci.md Vue interpolation build error (#975)
* rewrite deploying-apps as a lifecycle guide, refresh landing, comparison, troubleshooting, status (#974)
**Features**
* add 12 service templates (catalog tranche 5) (#683)
* add 16 new self-hosted service templates to catalog (#807)
* add Azure and GCP cloud node provisioning (#782)
* add dashboard toggle for HSTS (#840)
* add opt-in per-app deploy status badge (#898)
* add platform capabilities panel to empty dashboard (#870)
* add react-i18next foundation, migrate 3 deploy settings cards (#905)
* add standalone template detail and catalog routes (#814)
* add thesvg logo mappings for 4 catalog templates (#805)
* add whole-mesh network topology view (#819)
* adopt an existing machine as a node over SSH (#822)
* agent certificate renewal, agent-generated keys, re-enrollment and agent version reporting (#710)
* agent identity on API tokens and audit entries (#765)
* agent onboarding (init, agent identity UI, log levels, agents settings page) (#769)
* AI model deploys on GPU nodes, load balancer, S3/R2 log archive, pipelines, catalogue (#701)
* AI serving load (model page, engine metrics, VRAM fit, on-demand residency, CDI attach) (#770)
* app health timeline on the overview page (deploys, restarts, error windows) (#694)
* app quick actions menu, redeploy button, empty-state CTAs (#672)
* app timeline, pending changes, secret declaration on set, domains CLI and previous-release hold cap (#718)
* audit MCP tool surface and add a token budget test (#758)
* auto-pick a free dashboard port, make ingress ports overridable (#926)
* auto-route the status page's custom domain through the embedded ingress (#936)
* beta feature push (domains/DNS, SSL certs, volumes, NAS, VPN mesh, firewall, signatures) (#884)
* brand logos for frameworks, integrations and backup targets (#671)
* bring glinui tokens into the control-plane dashboard (#907)
* browsable template marketplace with category filters and search (#882)
* build node routing preference and node onboarding clarity (#775)
* cancel, queue and digest-pinned rollback for deploys (#731)
* **catalog:** add batch 3 of new self-hosted templates (#925)
* **catalog:** add batch 4 of self-hosted templates (#940)
* **catalog:** add batch 5 of self-hosted templates (#954)
* **catalog:** add YAML-loaded template path for contributors (#918)
* change correlation on alerts and SLO burn-rate rules (#742)
* clean no-root install path for levelrail-cli, like aws/gh (#889)
* cli build detect/branches and restore history lists (backups, pitr, app-volume-backups) (#684)
* **cli:** add global deploy freeze commands (#951)
* **cli:** wire up orphaned containers, network shares, and topology/traffic views (#945)
* cloud node provisioning (Hetzner, DigitalOcean) (#776)
* command palette actions, per-app quick actions, recent items and keyboard hints (#693)
* compact log query for agents (MCP query\_logs, CLI logs query) (#763)
* connect apps to managed databases via API, CLI, and UI (#818)
* connection-lost banner with backoff reconnect and 12 new service templates (#696)
* control plane backups card on general settings (#675)
* control plane self-backup and offline restore (snapshots, scheduler, pre-migration copy, downgrade guard, CLI, docs) (#678)
* cross-app deployments API with summary, stream, CLI and MCP tools (#725)
* cross-app deployments page with live feed, filters, drawer and shortcuts (#728)
* dashboard consistency pass (settings nav, search, headers, InfoTip docs links) (#838)
* dashboard redesign with a UI kit, overview v2, mission control home, visual apps list and grouped navigation (#715)
* dashboard setup checklist, deploy failure summary and keyboard shortcuts help (#699)
* dashboard waves 1-2, domains attention and backup relocation (#844)
* deploy preview screenshots (opt-in thumbnails per deploy) (#727)
* Docker Compose depends\_on start ordering, unsupported key validation, apps validate (#784)
* **docs:** add amber WebGL ambient field to homepage hero (#832)
* **docs:** give each landing feature card real proof, not a generic icon (#878)
* **docs:** redesign footer with scroll-reveal and glass pill accents (#871)
* **docs:** scope hero WebGL field to hero, add scroll parallax and footer stars (#834)
* doctor check for stale control plane backups, verified pre-upgrade snapshot and downgrade guard live (#685)
* encrypted off-box control plane backups, platform as code, alert silences and a public status page (#712)
* enlarge Levelrail mark, fix header's placeholder logo (#868)
* env editor .env import preview, pending diff, secret-safe export, and apps env import/export CLI (#690)
* failed deploys and disk pressure in attention (#673)
* fan out container metrics collection to remote node agents (#960)
* forward raw TCP streams through the embedded ingress (#921)
* GPU-aware scheduling, S3 build cache, pipeline graph and repo sync, fork PR guard, gateway fix (#702)
* harden and polish the in-app AI assistant chat (#899)
* harden PR preview environments (caps, fork approval, single comment) (#745)
* Hugging Face preflight and model cache manager (#741)
* import 36 Coolify-compatible catalog templates (devtools, productivity, automation) (#896)
* **integrations:** add free self-hosted catalog entries (#914)
* label certificate events in the audit log (#919)
* land beta-push waves 2-8 (cloud provisioning, scheduled deploys, preview DB isolation, pipeline OIDC, node doctor checks) (#962)
* load balancer check history, check now and per-upstream admin state (#719)
* load balancer page redesign with live topology, suggestions, health history and export (#721)
* local checks post PR signal, flag new comment bloat pre-commit (#862)
* local dogfood run, experimental gate cleanup, security alert verdicts (#773)
* lock Levelrail visual identity and wire it in (#867)
* log API request latency with slow/critical bands (#924)
* log controls, brand logos, status page, disk and cert guards (#663)
* log level tags, level filter chips, expandable rows and jump to first error (#669)
* machine-readable CLI errors and a --json coverage test (#762)
* manage orphaned containers, not just view them (#859)
* mark one-click template deploys as trials, add a stop-and-delete banner (#886)
* MCP tools for attention, node status history, audit search and control plane backups (#682)
* node\_offline alert kind (API, CLI, dashboard, docs) (#691)
* observability settings, smarter health checks, Ask AI, test perf (#922)
* one-click deploy for templates with no required config (#811)
* overnight batch - docs, domain collision guard, ingress-aware placement, traffic screen, db backup fixes (#938)
* overnight batch (cloud node provisioning, real 2-node verification, scheduled deploys, agent/MCP tooling, templates, and more) (#792)
* overnight batch 2 (compose replicas, scheduled-deploy verification, pipeline OIDC, Redis preview isolation, node doctor checks) (#801)
* passkey login, Microsoft OAuth, Resend email, and a redesigned login screen (#855)
* path filters, merge queue trigger, and forge status reporting for pipelines and deploys (#744)
* per-key daily token budget, created\_by, revoke MCP tool and reworked model keys panel (#740)
* platform ops load (apps metrics batch, promote/clone plans, safe upgrade, move plan, honest comparison) (#768)
* readyz endpoint, CLI parity tranche 2, MCP backup verify and failed deploys tools, e2e cleanup (#700)
* real preview thumbnails on the deployments page and cross-app API (#737)
* rebuild homepage sections to drop templated card-grid feel (#872)
* redesign docs site with custom nav/sidebar and AI-readiness (#860)
* redesign setup wizard server check into a compact summary (#821)
* redesign the git repo picker as a searchable card grid with provider tabs (#935)
* replace keyboard shortcuts dialog with a stage-overlay nav surface (#908)
* report certificate renewal state (ok or stalled) in API, CLI and dashboard (#670)
* request metrics, deploy safety, model keys, app management, import, preflight and migration runner fix (#711)
* roll the real mark out to every remaining fallback spot (#873)
* rootless Docker/Podman detection + pipeline OIDC federation (#785)
* search public Docker Hub from the docker-image deploy picker (#815)
* security follow-ups, control plane backup stale alert and verify, generated API reference (#697)
* server-side audit log search and failed-only filter (#674)
* settings sidebar UX, readable in-app docs, CLI --debug/pagination, 16 more catalog templates (#912)
* show WAF, redirect, and maintenance status on the Domains page (#843)
* sidebar visual redesign with a real expandable Projects tree (#857)
* skeleton-first loading states across remaining detail and settings routes (#847)
* SLO burn-rate auto-rollback with auto, dry-run, and pause-for-human modes (#810)
* structured deploy failure object across API, CLI and MCP (#761)
* supply chain visibility per deploy (SBOM, optional scan gate) (#743)
* support connecting multiple GitHub orgs to the GitHub App (#958)
* support MariaDB in the slow query log viewer (#858)
* surface cross-node ingress unreachability instead of failing silently (#933)
* surface IAM/registry in top nav, real skeletons on detail pages (#842)
* tiered deploy previews with a free metadata default (#736)
* **toast:** wire navigable actions into high-value toasts (#952)
* unify brand to petrol-blue, redesign docs homepage, bring glinui tokens to web (#909)
* unify wizard step chrome across setup and node wizards (#849)
* update channel settings, scheduled check, and version-skew notice (#866)
* wait\_for\_deploy and plan\_change for the agent layer (#766)
* wave 3 (changelog, API explorer, cost estimator, templates, push, env diff, topology, forecast, chat approvals) (#890)
* wave 3 agent surface (experimental gate, failure classes, failure view, agent loop e2e, launch docs) (#772)
* **web:** add brand logos for 5 of the selfhosted4 templates (#812)
* weekly GHCR cleanup for orphaned untagged image versions (#864)
* wire queue, cancel and digest rollback into the deployments page (#750)
* wire up [support@levelrail.com](mailto:support@levelrail.com) and [support@glinr.com](mailto:support@glinr.com) contacts (#827)
**Bug Fixes**
* \[HIGH] ssrf in http log drain sinks (#894)
* \[HIGH] ssrf in http log drains (#688)
* accessibility and baseline-ui pass on the beta feature wave (#887)
* accessibility pass on status, log viewer and node events, plus component tests (#681)
* acknowledge GitHub ping events on the git webhook route (#932)
* add bearer-compatible /auth/whoami so auth whoami works with API tokens (#966)
* address SonarCloud gate findings on overnight2 (#804)
* bound agent gRPC GracefulStop so upgrades stop blocking ingress (#943)
* bridge managed databases onto an app's own network (#937)
* cache-bust the favicon so a redeploy actually shows the new one (#877)
* cap API token abilities to the caller's own, add route guard test (#689)
* **catalog:** correct pairdrop image tag, 1.19.0-ls94 does not exist (#961)
* change correlation and SLO burn review follow-ups (#746)
* close a status race in SSH node provisioning (#863)
* collapse long always-visible prose into on-demand InfoTip (#903)
* confirm dialog for deploy cancel, aria-hidden on status header icon (#895)
* CPU percent always reads near zero on some Docker installs (#904)
* dashboard review follow-ups (health link, fallback url, unhealthy diagnosis, cleanup card, read on open, lazy row metrics, stop polling on 404/501) (#722)
* dashboard visual polish across metric tiles, node providers, and domains (#856)
* deploy preview review follow-ups (opt-out, deletion, labeling races) (#733)
* deployments page review followups (pinned redeploy, confirm copy, deep link, retry loop) (#730)
* deployments preview follow-ups (#739)
* deployments review follow-ups (per-event visibility, old held in needs\_attention, rollback and live precision) (#726)
* detect actually-open dialogs, not closed-but-mounted ones (#820)
* Docker web-build symlinks, stop forcing full CI on every main push (#911)
* **docs:** footer giant mark position, mobile overflow, and scroll reveal (#875)
* **docs:** keep hero stars twinkling instead of fading to black, pause offscreen (#874)
* drop --delete-branch from dependabot auto-merge, incompatible with the merge queue (#906)
* extend pre-commit comment-density check to web/\*.ts and \*.tsx (#910)
* force vite to apply CJS interop to mermaid's fastdom dep (#831)
* gate ingress routing and deploy-attempt status on real readiness (F-002) (#774)
* GitHub App manifest redirect blocked by its own CSP (#929)
* hugging face preflight review follow-ups (docker disk, per-GPU fit, gated ollama, file lookup, cache key, UI) (#747)
* IaC env placeholders require an explicit allowlist and triage of open CodeQL alerts (#717)
* IAM scoping for app routes, review findings, load balancers and pipelines overview pages, salvaged Jules tests (#708)
* judge deleted-app approvals by IAM instead of hiding them (#732)
* launch wave 1 (nightly, CodeQL, agent-core MCP profile, env tools, VPS smoke) (#759)
* load balancer page review followups (safe export commands, no default probe, retry 0, serialized saves, estimated shares) (#723)
* make update preflight checks respect the configured channel (#942)
* mesh placement bug, safe SQLite volume backups, settings sidebar (#880)
* node provisioning follow-up fixes from post-merge review (#779)
* paint the hero WebGL field's first frame unconditionally (#833)
* passkey login backup-eligible flag (supersedes #941) (#947)
* pin the e2e PITR minio image and skip when the registry is unavailable (#698)
* pipeline script injection, IAM and stream re-auth gaps, secret binding, gateway limits, MCP safety, container hardening, signed releases (#709)
* point MinIO template at its still-public registry path (#913)
* preview hardening and agent-core profile follow-ups (#771)
* preview tiers review follow-ups (card kept on failure, public-domain Host, bounded thumbs) (#738)
* read host memory on macOS and report unsupported platforms cleanly instead of a raw /proc error (#714)
* recover two commits orphaned by an earlier merge-queue timing race (#865)
* reject a deploy or rollback while one is already running (#934)
* reject a second concurrent manual build of the same app, add deploy pipeline failure-mode tests (#692)
* reject noncanonical upstream ids and clear admin state when a load balancer is removed (#720)
* remove duplicate backup\_history index migration (#788)
* remove duplicate backup\_history migration and quote colons in docs frontmatter (#790)
* remove unresolved merge markers from main's api-reference.md (#900)
* render README badges, add logo, restore merged README structure (#967)
* renumber the deploy approval options migration to 0144, it collided with the status page migration at 0142 (#716)
* replace window\.confirm with the standard delete-confirm dialog (#800)
* resolve database env vars to mesh DNS names, not container names (#816)
* resolve migration 0135 number collision (#787)
* resolve non-default branch refs in manual build triggers (#930)
* reuse existing badge variant, add inferable name defaults (#797)
* reuse stored git-source token, merge ingress settings on set (#928)
* review follow-ups for path filters and forge status reporting (#748)
* scope cross-app list endpoints to apps the caller can read (#734)
* scope failed deploys and deploy approvals lists to readable apps (#729)
* scope lint cache per worktree, harden AI-attribution check (#793)
* security and correctness findings from review of #711 (#713)
* security hardening (log redaction, login timing, CSP) (#885)
* signal agents to close their session stream on shutdown (#953)
* silence shellcheck SC2016 false positive in install.sh test (#789)
* sort beta releases by publish time, not list order (#931)
* static-site CLI gaps, framework detection, webhook URL, git import (F-017/F-018/F-020/F-021/F-022/F-023) (#778)
* stop forcing the favicon mark to look small (#876)
* stop rebuilding Docker images on every push to main (#946)
* stop reporting a cross-node domain app as Healthy (#948)
* stop showing a dead container's logs as live in blue-green overlaps (#923)
* stop showing add-tag control on every app tab (#920)
* stop squeezing the template catalog into a narrow fullscreen column (#845)
* supply chain gate review follow-ups (#749)
* unbounded scoped backup paging and case-insensitive cert domain ownership (#735)
**Performance**
* batch environment lookups in GET /api/v1/apps (#955)
* batch SQLite telemetry inserts using json\_each (#881)
* batch TLS secret lookups in GET /api/v1/databases (#956)
* cover webhook delivery pagination with a composite index (#957)
* idle footprint benchmark, cached Docker disk usage, single container list per suspended reconcile (#695)
* index backup\_history by started\_at (#705)
* optimize get conditions for controllers query (#791)
* optimize ListAppEvents and ListDeployFreezeWindows IN clauses (#893)
* Skip weekend run (#724)
* split internal/api test shard from 3 to 4 in fast CI lane (#795)
* use json\_each for SQLite IN clauses (#802)
**Documentation**
* add a serif pull-quote accent card to the homepage (#836)
* add a who-this-is-for page grounded in shipped features (#915)
* add credibility content to homepage (#835)
* add feature status page with per-area evidence and labels (#756)
* add full template catalog page (#916)
* add metallic headline gradient to homepage hero (#829)
* add real mermaid architecture diagrams to 16 pages (#852)
* apply the pending corrections from feature-status.md's audit (#917)
* capture and embed load balancer, app overview, and network screenshots (#854)
* clarify the dashboard port isn't always 8080 (#927)
* embed dashboard screenshots across guides (#965)
* explain mesh DNS resolution for cross-node database connections (#817)
* fill network topology and connections gaps, tighten onboarding (#825)
* fix llms.txt domain, add sitemap lastmod/priority, per-page OG image (#837)
* lead git-integrations, templates, feature-flags, and projects pages with user tasks (#853)
* lead with user tasks instead of implementation details (#841)
* new Accordion/Steps/Tabs/InlineToc components, trim filler prose (#944)
* note APP\_AGENT\_ADVERTISE\_HOST requirement for real node enrollment (#780)
* polish homepage cohesion across tonight's content passes (#839)
* record the docs site's amber accent as a deliberate second brand color (#879)
* redesign homepage with bento feature grid and real visual identity (#828)
* refresh roadmap, feature catalog, README status and CLI reference (#676)
* remove GitHub star count from homepage trust strip (#846)
* render mermaid diagrams, search body text, rewrite operator-facing content (#777)
* rewrite README, getting started, installing, architecture, docs index (#964)
* serve install.sh from levelrail.com instead of raw GitHub URL (#830)
* strip inline implementation names from resilience, DR, and backups pages (#851)
* sync template catalog count to 311 everywhere (#968)
**Bug Fixes**
* resolve three-way migration 0119 collision and a test fake left stale after a parallel merge (#659)
**Features**
* branch-scoped env var overrides for preview deploys (#632)
* per-app integrations catalog (PostHog, Sentry, Datadog, and more) (#647)
* pre-flight framework detection and live deploy step feed (#648)
**Features**
* unblock Python/Django Railpack deploys (#623)
**Bug Fixes**
* harden Vikunja and Keycloak catalog templates toward production defaults (#622)
**Bug Fixes**
* sort install.sh release list by created\_at, not API order (#619)
**Bug Fixes**
* correct Vikunja and Keycloak catalog templates so they boot (#615)
**Bug Fixes**
* bundle docs/ into the web build stage of the Docker image (#611)
**Features**
* https, redirect, status-range and exec health probes; active checks for 71 catalog templates (#610)
* rich release notes, release announcements, and an SEO changelog on the docs site (#607)
**Bug Fixes**
* correct install/compose docs and fix doctor port\_80 false fail (#591)
* resolve high and critical CodeQL alerts (SSRF, email injection, zip slip, path injection, enrollment CA pin) (#609)
**Features**
* setup wizard with live server, domain, git, and first app verification (#587)
**Bug Fixes**
* bump dependencies for security advisories (#583)
**Features**
* actionable CTAs on System Status, reusable docs help links (#551)
* add container restart count and build duration metrics (#560)
* clone a whole environment's app set into a new environment (#562)
* configurable ingress HTTP/HTTPS listen ports (#556)
* detect and clean up orphaned named Docker volumes (#569)
* distribute WireGuard mesh config to remote nodes over gRPC (#564)
* doctor network, resource, and multi-firewall checks (#581)
* egress allowlist UI on the app network page (#545)
* empty states with prerequisite hints and template CTA (#579)
* fleet-wide node CPU/memory/disk utilization view (#575)
* Gitea as a fourth git provider (#553)
* healthchecks and memory advisories for all 129 catalog templates (#576)
* install works end to end (setup token, HTTP login, installer preflight) (#582)
* per-app exec/shell access opt-out, defaults enabled (#542)
* per-app outbound network allowlist (egress gateway) (#543)
* point-in-time restore for Postgres databases (#566)
* secret age tracking and rotation reminders (#563)
* slow query log for Postgres/MySQL databases (#565)
* two-person approval gate for deploys into protected environments (#567)
* WireGuard mesh key rotation and peer status visibility (#561)
**Bug Fixes**
* app status rollup, compose service pruning, and group lookup by app name (#552)
* confirm detach actions and surface missing success toasts (#546)
* detect frozen agent processes via heartbeat frames and gRPC keepalive (#558)
* doctor's disk space check shows raw byte counts, not human-readable (#550)
* live resource updates fail silently when only memory is set, no swap (#548)
* rate limit the unauthenticated git webhook receiver (#574)
* retry transient SMTP failures for email notification channel (#559)
* stop doctor's port 80/443 check from false-failing on its own ingress (#547)
* stop treating unlabeled Docker networks as this instance's own (#557)
* type errors blocking the real production build (npm run build) (#549)
* wire Bitbucket App secrets manager, OAuth connect was permanently broken (#555)
**Features**
* CLI coverage for backup download, database logs, node placement (#529)
* CLI coverage for database attachment, manual build trigger, moves, resources, AI settings, git providers (#536)
* per-service readyTimeout override for the reconciler's readiness budget (#539)
**Bug Fixes**
* accessibility gaps in form labeling and muted-text contrast (#528)
* classify a pre-port-bind crash as a crash reason, not generic ReadinessFailed (#535)
* enforce required secrets on the build-triggered deploy path, not just webhook (#541)
* extend IAM resource scoping to 79 mutating app and database routes (#532)
* make tag CLI commands consistently take a name, not a mix of name and ID (#533)
* manual builds tag by resolved commit, pending image never hits Docker (#530)
* never treat a master key read failure as no key yet (#522)
* nudge reconciler after a build-triggered deploy completes (#534)
* rearm crashloop auto-rollback for a second, different bad deploy (#540)
* resolve migration number collision at 0106 (#524)
* respect prefers-reduced-motion and avoid layout-triggering progress transition (#531)
* scope container and network cleanup to this control plane's own instance (#537)
**Features**
* add curated local AI model templates to the service catalog (#514)
* add domain-level custom error pages (#511)
* collect real host memory for the control plane's own node (#515)
* live RAM-fit check in the AI template wizard (#516)
**Features**
* add concurrency\_policy to scheduled tasks (#495)
* add deploy log download endpoint, CLI command, and UI button (#504)
* add DNS-01 provider abstraction with Route53 as a second provider (#492)
* add domain-level redirects (#507)
* add instance-wide backup history view (#503)
* default published ports to a private bind address (#496)
* per-preview env var overrides on parent app (#493)
* per-service pull\_policy support for Docker Compose imports (#494)
**Bug Fixes**
* lazy-load scoped sidebars, align react-dom to react 19.3.0 (#499)
* renumber colliding scheduled-task migration to 0101 (#506)
* resolve migration number collision at 0098 (#505)
* restore explicit Control cast for BackupMissingFields props (#500)
**Features**
* add apps wait and apps deploys list CLI commands (#460)
* add backup\_missing alert rule kind (#463)
* add Content-Security-Policy and opt-in HSTS, fix healthz routing (#449)
* fail fast when a container is OOM-killed during readiness wait (#451)
* first-party GitHub Action for deploying to Levelrail from CI (#461)
* move an app to another node with its volumes (#464)
* resolve app secrets live from an external HashiCorp Vault instance (#465)
* selective backup archive deletion (#462)
* zero-config sslip.io URL for apps with no domain configured (#447)
**Bug Fixes**
* bound live Docker inspect calls with a timeout (#450)
* cut new-code duplication in the move-with-volumes handler and CLI (#466)
* poll deploy status live so a rollback's actual progress is visible (#452)
* recognize every reconciler failure reason in the rollout stage (#458)
* retry transient notification delivery failures (#454)
* revert cel-go bump that broke the build (#459)
* suppress duplicate lines when a log stream's EventSource reconnects (#453)
**Features**
* add bulk restart for every app in a project (#442)
* add database stop/start lifecycle, API, CLI, and dashboard (#440)
* pause and resume an entire project (#441)
* production-readiness batch - reconciler responsiveness, hardening, README (#445)
* rank apps by live CPU, memory, and network usage on the dashboard (#443)
**Bug Fixes**
* recognize AlreadyRunning as a done rollout condition (#439)
**Features**
* add CLI metrics query and live log tailing (#394)
* add VitePress docs and landing site for GitHub Pages (#387)
* alert/channel editing, deploy-notify+promote MCP, secrets at app-create, reconciler test gaps (#433)
* close CLI feature-completeness gaps for settings, storage, and git apps (#396)
* compose command/bind-mounts, self-service invites, more templates/tools/channels, e2e hardening
* compose entrypoint, app.yaml command/bind-mounts, external registry browsing, more templates and MCP tools (#385)
* dispatch builds to a remote build node over the agent transport (#432)
* ephemeral database per preview environment (#397)
* interactive exec terminal over WebSocket (#436)
* liveness probing and automatic restart (#434)
* node-move teardown fix, simple spread scheduling, CLI/MCP completeness batch (#395)
* opt-in per-domain WAF and rate limiting for embedded ingress (#398)
* remote exec and Docker networking over the agent transport (#399)
* replace guesswork form fields with real pickers and suggestions (#437)
* TLS for managed Postgres and Redis connections (#364)
**Bug Fixes**
* address containerd and docs-site dependabot alerts (#392)
* chain release.yml to actually run after release-please tags a release (#373)
* derive Select trigger label automatically from item children (#438)
* make manual release.yml dispatch actually target the right tag (#375)
* pin cel-go to v0.28.1, restoring build compatibility with caddy (#389)
* raise Docker image build timeout and add layer caching (#378)
* route error components against unknown, not Error (#431)
* spread drained resources across nodes, warn on dropped depends\_on (#435)
* stop and remove containers when a service is deleted (#381)
**Performance Improvements**
* build Docker images on native per-arch runners instead of QEMU (#380)
**Features**
* actionable restart-required toast for health check and resource limit saves
* add a restart action for apps
* add age-based backup retention alongside count-based retention (#194)
* add backup targets settings page
* add build.base\_directory for monorepo subdirectory builds (#139)
* add CLI support for deploying a Docker Compose file (#109)
* add cloudflare-tunnel CLI subcommands (#188)
* add CPU/memory quick-stat cards to app overview (#163)
* add dashboard home page at /
* add developer-view raw text editor for env vars
* add Dokploy as a migrate CLI source (#135)
* add Dragonfly and ClickHouse as managed database engines (#145)
* add editable port control to app overview page
* add environment-level shared env var tier between project and app (#195)
* add fullscreen size variant to dialog primitive
* add fullscreen toggle to the create resource wizard
* add GET .../secrets and POST .../secrets/\{key}/lock endpoints
* add interactive wizard mode to apps create (#239)
* add interactive wizard mode to databases create (#245)
* add levelrail-cli doctor command for local preflight health checks (#265)
* add list\_audit\_log MCP tool (#285)
* add loading states across dashboard routes and data-fetching components (#248)
* add MariaDB and KeyDB managed database engines (#141)
* add MCP server, extract shared internal/apiclient package (#121)
* add migrate caprover CLI command (#191)
* add Network tab showing live traffic path from domain to container (#148)
* add nodes CLI subcommands (#192)
* add non-interactive exec endpoint for running commands in app containers
* add organization-level shared env var tier above projects (#193)
* add periodic OS package-update check per node (#220)
* add projects CRUD, project env vars, and project membership to the CLI (#250)
* add reversible per-secret lock to the store layer
* add secret key listing and lock mutation hooks
* add settings sub-sidebar and fix stale route tree (#222)
* add SetValueGuarded, ListKeys, SetLocked to secrets.Manager
* add structured issue templates and type-specific PR templates (#77)
* add System status dashboard page (#324)
* add test-connection action for backup targets (#307)
* admin recovery email, general email capability, forgot-password flow
* admin recovery email, general email capability, forgot-password flow (backend)
* alert on repeated scheduled task failures (#290)
* allow clickhouse as an app.yaml database engine (#186)
* allow pinning a service's host port (#170)
* App Clone
* app detail screen polish - deploy history, grouped nav, build hints, convergence
* App Overview hero+checklist, batched Apps list status
* app volume backups (backend, API, CLI, and web UI) (#311)
* app-detail polish - deploy columns, grouped nav, convergence badge, build-log hints
* app-level scheduled tasks (#140)
* apply resource limit changes to running containers live (#242)
* apps deploy/status/logs and databases commands for the CLI
* apps restart/rollback CLI subcommands
* apps table + read-path for multi-service app support (stage 1) (#61)
* attribute audit log entries to their client and add retention (#316)
* audit log CSV export via API, dashboard, and CLI (#277)
* audit log for write/deploy/root-tier requests (#125)
* auto-generate and persist master key on first boot (#60)
* auto-verify scheduled backups right after they complete (#280)
* backup file download to browser
* backup integrity verification (#273)
* backup target onboarding, connect an S3-compatible bucket
* bounded health check and opt-in ufw setup in install.sh (#323)
* breadcrumb and hierarchical navigation for org/project/environment/app (#173)
* browse and deploy from the curated service-template catalog (#119)
* build Docker Compose services with build: directives (#190)
* build pack selector and branch picker for git app creation
* build.type: image, deploy a prebuilt registry image with no build (#124)
* built-in container registry (#366)
* BYO TLS certificate upload for domains (#363)
* centralized Domains page with ACME settings and app-domain list
* changed-line coverage gate for PRs (#350)
* CLI and dashboard wiring for organizations and environments (#169)
* CLI CRUD for backup targets and registry credentials (#251)
* CLI device login flow (auth login --device) (#319)
* CLI exec, auth login/whoami, tokens, domains, backups commands
* CLI exec, auth, tokens, domains, and backups commands
* CLI output formats and JMESPath query filtering (#318)
* CLI support for app-level scheduled tasks (#144)
* ClickHouse and Dragonfly backup/restore support (#157)
* close git provider capability gaps (GitLab branches, GitHub webhooks, aggregated status) (#228)
* Cloudflare Tunnel integration for outbound-only ingress (#138)
* collect and surface node disk space as a real telemetry metric (#185)
* connect a GitHub App for private-repo access and repo browsing
* containers page and master key rotation UI, feature catalog doc (#358)
* Coolify migration CLI (levelrail-cli migrate coolify) (#45)
* copyable DNS record rows and better layout for domain check (#63)
* curated named roles (admin/operator/viewer) over the ability model (#240)
* curated service-template catalog and read-only browsing API (#117)
* custom Docker labels escape hatch on spec.Service
* dashboard empty states for alerts, deploy history, and log search (#238)
* dashboard home page at /
* database backups to S3-compatible storage, with restore
* database publicly-accessible toggle + host port
* delete pruned scheduled-backup objects from their bucket
* deploy comparison view (#272)
* deploy-outcome notifications
* deterministic resource right-sizing suggestions (#279)
* dismissible dashboard nudge to enable platform-wide alert rules (#300)
* docker compose as a deploy target (#97)
* draft autosave and guided defaults for create-resource forms (#260)
* editable port control on app overview
* empty states and inline field guidance
* enforce frontend bundle size budget in CI (#299)
* env var tier provenance and audit activity panel in EnvEditor (#261)
* env/secrets developer view (frontend)
* expand MCP server tool coverage to nodes, previews, alerts, metrics, rollback (#252)
* explicit Stop/Start action for apps
* extend resource right-sizing recommendations to databases (#288)
* fan out git push webhook deploys to sibling app-group services (#172)
* feature flags with live evaluation via existing API tokens (#262)
* fill CLI gaps for app lifecycle, database delete, secrets, git-source (#247)
* forgot-password flow and email settings pages (frontend)
* forward app logs to an external HTTP or syslog sink (#137)
* framework-aware guided deploy picker
* full-screen presentation for the create resource wizard
* fullscreen log viewer toggle and finish-aware auto-scroll (#226)
* general per-actor rate limit on the authenticated API surface (#284)
* generic OIDC SSO as a third login provider (#160)
* GitHub Enterprise Server and Bitbucket Cloud git provider support (#221)
* GitLab App integration (#127)
* global notification channels settings page
* guided domain add flow with inline TLS status
* guided domain DNS check
* guided first-run onboarding flow for zero-app instances (#266)
* HTTP Basic Auth protection for domains, with helpful no-domain UX (#162)
* IAM policies dashboard page (#322)
* IAM-style resource-scoped policy engine (#317)
* implement database dump mechanism for backup targets
* implement database restore from backup
* implement Redis backup restore
* implement rolling deploy strategy (#171)
* ingress settings and domains API endpoints
* ingress\_settings store for ACME toggle and primary domain
* land on the live build log after triggering a build (#67)
* let operators pick between CNAME, A, and AAAA for domain setup (#189)
* levelrail-cli version command and CHANGELOG (#301)
* lightweight non-RBAC Projects grouping
* list every container on a node, not just Levelrail-managed apps (#326)
* list secret keys with reversible lock toggle in SecretsEditor
* live per-node alert status for patch\_status, node\_disk\_space, node\_resource\_usage (#304)
* live SSE log broadcaster and stream endpoint for app containers
* live-tailing terminal viewer for app container logs
* log export as a downloadable text file (#276)
* make the New resource wizard friendlier to non-technical users (#70)
* managed database logs and metrics (#225)
* manual Docker disk/image/volume pruning
* manual GitHub App connect flow, domain disclaimer on automated setup (#104)
* master key rotation for envelope-encrypted secrets (#295)
* MCP tools for feature flags, doctor, webhook deliveries, backup verification, deploy compare, notification deliveries (#274)
* memory/CPU resource limits for databases
* MongoDB as a managed database engine (#120)
* multi-service apps stage 2 (deploy fan-out + per-app Docker networking) (#75)
* multi-service support in apps create --interactive (#349)
* multi-user sessions, OAuth sign-in (Google/GitHub), users management
* multi-user sessions, OAuth sign-in, users management API
* named credentials profiles (--profile / APP\_PROFILE) (#315)
* named-volume support for ordinary application services (#76)
* node CPU/memory alert rule (node\_resource\_usage) (#297)
* node-level resource dashboard
* non-interactive one-off exec in app containers
* notification channel test-send delivery history and CLI/UI wiring (#267)
* OAuth sign-in UI, OAuth and Users settings pages
* PagerDuty and Microsoft Teams notification channels (#151)
* paginate backup execution history (#143)
* per-domain maintenance mode (#360)
* per-user RBAC, extending the existing ability system to sessions (#131)
* persisted per-app git source + multi-app GitHub webhook
* persistent restart-required signal for env var changes (#263)
* post GitHub PR comments and commit statuses for preview deploys (#361)
* pre/post-deploy hook commands (#362)
* preview environments per pull request (#246)
* proactive alerting for certificate expiry (#283)
* proactive domain\_health alert rule kind (#308)
* proactive node disk space alerting (#293)
* proactive node patch status alerting (#289)
* promote an app's image across environments in the same project (#278)
* protected environments with confirm-to-deploy friction gate (#282)
* Pushover notification channel (#126)
* Railpack auto-detection for Node.js and Go builds
* read-only dashboard visibility for static sites
* read-only deploy failure and crashloop diagnosis (#268)
* real ACME issuer option in ingress config builder
* real ACME issuer, platform primary domain, and centralized Domains page
* real app-to-database attachment (#146)
* real deploy history, rollback, and build-log persistence
* real multi-marker deploy history on the metrics chart
* real S3-compatible upload client for backup targets
* real-time deploy stage timeline and in-progress banner (#158)
* real-time live log streaming for app containers
* redesign deploy detail page with metadata card and collapsible stages (#165)
* redesign New Resource picker into a categorized, searchable modal (#110)
* registry credentials for pulling private images (#130)
* relabel git-source wizard card, elevate Railpack, add connect-GitHub CTA and domain field (#68)
* replace one-off empty states with shared EmptyState component (#164)
* replace single-admin store with multi-user users table
* required-check CI gate, CodeQL path filtering, label taxonomy and auto-labeler (#95)
* resolve SERVICE\_FQDN\_\* magic vars to a service's own assigned domain (#113)
* resolve template SERVICE\_ magic vars in compose deploys (#108)
* restore a backup into a brand-new database, alongside in-place restore (#310)
* restore app volume backups as a new, standalone volume (#313)
* reuse GitRepoSourcePicker in GitSourceCard (#229)
* scheduled database backup configuration UI and CLI (#174)
* scheduled/cron backups
* scope new-resource picker to the caller's intent (#254)
* secret key list/lock API + scope deploy trigger card to Overview/Deploys
* service templates, team invites, deploy comparison, notification channels, and MCP tools
* settings hub, global search, updates page, and ingress DNS transparency (#111)
* shared/group env vars at the project level
* shell completion for levelrail-cli (bash, zsh, fish) (#270)
* splice generated secrets into composite env values (#118)
* storage as an app resource (R2/S3)
* surface compose restart: and networks: as non-blocking notices (#287)
* surface deploy strategy and replicas on the app screen
* surface Docker daemon reachability as a dashboard-wide health check (#259)
* swap memory and CPU pinning (cpuset) resource limits (#233)
* test registry credentials before deploy relies on them (#291)
* thread ingress settings into the ingress reconciler
* TOTP two-factor authentication for admin login (#114)
* track commit sha and trigger source on deploy attempts
* translate compose healthcheck blocks into readiness probes (#286)
* trigger and track a real database backup
* trigger and track database backups from the database page
* unify git provider connect flow, fix missing git-source and build bugs (#227)
* unify webhook-triggered and manual multi-service deploys (#241)
* v1 install script, systemd unit, release build workflow (#44)
* webhook delivery history and replay (#269)
* wildcard domain support via Cloudflare DNS-01 (#152)
* wire alert rules to global notification channels
* wire developer-view toggle into the Environment section
* wire Dockerfile build args through app spec, API, CLI, and UI (#155)
* wire IAM policy resource-scoped enforcement into apps/databases routes (#321)
* wire mesh DNS server into container creation
* wire OAuth secrets, update recover-admin for multi-user store
* wire prebuilt-image build type into CLI, UI, and git-source validation (#129)
**Bug Fixes**
* accessibility gaps in git picker, log terminal, and nav sidebars (#232)
* accessible names for placeholder-only domain/env form inputs
* adapt notification\_channel.go to the email.Sender interface
* add accessible names to per-row feature flag toggle and rollout input (#275)
* add missing --json support to three CLI clear commands (#271)
* add TTL fallback sweep for preview environments across all git providers (#281)
* allow railpack build type through apps create (#147)
* async build trigger, private repo auth, and ability-gated token minting (#64)
* avoid BuildKit MergeOp deadlock in Railpack deploy image assembly (#149)
* clear stored secrets when disconnecting a GitHub App
* close forgot-password rate-limit bypass, remove dead admin-email frontend code
* close no-auth confirmation gap for KeyDB/Dragonfly, polish secrets CLI help (#303)
* close pre-existing AppDetail test-fixture typecheck gaps (#264)
* correct misleading test name/comment on a secrets test
* crashloop alert's view-logs link goes to a dead same-page anchor (#218)
* don't show a redundant inline error while the overwrite-confirm dialog is open
* drop stale preview-envs docs claim, skip live Docker tests in short mode (#314)
* eliminate race in TestHandleTriggerBuild\_RecordsDeployAttempt\_Succeeded (#298)
* explain why the live logs tail is empty, don't just wait forever (#161)
* feed persisted build logs to diagnose for build-phase failures (#292)
* fix overflowing create-resource picker, add category filter (#255)
* force 0.1.0 as release-please's first proposed version (#372)
* gate one-off container exec at AbilityRoot, not AbilityDeploy
* generate routeTree.gen.ts before web lint, trim test duplication (#116)
* get the main frontend bundle under the 500kB warning threshold
* guard ClickHouse restore live test against entrypoint's temp-server handoff (#312)
* image-sourced deploys show a clear no-build-output message
* image-sourced deploys show a clear no-build-output message instead of hanging on 'Waiting for log output...'
* land signed-in users on the dashboard, not /apps (#66)
* live deploy log stream stuck on zero output
* make password-reset token single-use atomic, tidy go.mod, trim comments
* mark orphaned deploy attempts as failed on startup (#159)
* move showDeployTrigger's useRouterState above the early-return
* pin cel-go back to 0.28.1, incompatible with pinned Caddy version (#134)
* pin golangci-lint-action to a commit SHA, suppress a false-positive Sonar finding (#352)
* point install.sh at the real first-run registration path (#325)
* poll for post-stop/remove state on CI's daemon
* poll for stopped state in Stop/Remove live test
* poll for stopped state in Stop/Remove live test instead of asserting immediately
* pre-fill and suggest commands in the exec panel (#327)
* raise internal/api CI test timeout to fix false-failure panics on main
* raise internal/api's CI test timeout to 25m
* re-ensure a container's network before restarting it (#150)
* record a deploy attempt when an app is created with a real image (#154)
* record deploy attempts for compose deploys and image-changing updates (#256)
* recreate a container whose Start fails even after EnsureNetwork (#153)
* reduce SonarCloud duplication flagged after #228 merged (#231)
* reject ACME contact emails with a display name
* renumber cloudflare-tunnel migration to 0049, collided with 0047 (#142)
* renumber deploy\_attempts\_commit\_source migration to 0025
* renumber domain-basic-auth migration to 0053, collided with 0052 (#166)
* renumber github app connection migration to 0029
* renumber ingress\_settings migration to 0024
* renumber migration to 0028 (0027 claimed by service\_labels)
* renumber migration to 0030 (0029 claimed by service\_git\_sources)
* renumber migrations past 0030\_service\_storage\_target.sql
* renumber oidc-provider migration to 0054, collided with 0053 (#168)
* renumber secret-lock migration to 0038 (0037 collision with PR #56)
* renumber users migration to 0035, dedupe nullableString helper
* require MySQL readiness probe to hold twice before proceeding
* resolve duplicate migration version 0052 (#167)
* resolve migration 0030 number collision (github-app vs storage-target)
* resolve migration 0030 number collision between github-app and storage-target
* restore docker/docker + go-connections to versions internal/build/build.go's ImageLoadWithQuiet call needs (#100)
* restore frontend checks in the pre-commit hook (#72)
* restore main build and resolve SonarCloud findings on multi-service apps (#99)
* restrict git branch listing to http/https schemes
* revert docker/docker, docker/go-connections, cel-go bumps (#98)
* revert typescript to 6.0.x, main is broken for npm ci (#96)
* rework forgot-password and admin-email for the multi-user model
* satisfy golangci-lint (revive, unparam) and fix dangling doc refs
* scope deploy trigger card to Overview and Deploys sections
* scope MongoDB restore to drop non-system databases first (#309)
* set explicit Content-Type on remaining webhook-derived response bodies (#249)
* small UX polish bundle (git-source validation, stale doc comment, database status dots) (#223)
* stop bundling zod into the eager main chunk via reset-password validateSearch (#230)
* stop clipping browse-templates category headers in a nested scroll box (#257)
* subscribe GitLab and Bitbucket webhooks to pull request events (#253)
* suppress SonarCloud false-positive on the intentional unconditional backfill UPDATE (#132)
* surface swap and CPU pinning limits on the app overview (#234)
* TestValidateDatabaseResource's unknown-engine case used mongodb, now a real engine (#122)
* unlock build fields after a failed build trigger in git app wizard (#224)
* unstick finished-attempt deploy log SSE connection status
* update Docker client call sites deprecated by the docker/docker 28.x bump (#103)
* use localeCompare when sorting env var keys in dev view (#351)
* verify github app installation status live against github, not just the local row (#294)
* warn on missing secrets master key before database creation
* widen TestCertStorage\_TouchLock\_KeepsItFresh's timing margin (#356)
* wire Dragonfly into app.yaml spec and database URL resolution (#187)
**Performance Improvements**
* scan only new log lines for build-log hints
# Chat-interactive deploy approvals
Source: https://docs.levelrail.com/chat-deploy-approvals
Approve or deny a protected-environment deploy directly from the Slack or Discord notification, without opening the dashboard.
When a deploy or promotion targets a protected environment, Levelrail requires a second, distinct user to approve it (see [Deploy safety](/deploy-safety) for the two-person gate). By default an approver opens the dashboard's Approvals queue. This feature adds Approve and Deny buttons to the Slack or Discord message Levelrail sends when a request comes in, so an approver can decide from a phone.
It is an opt-in addition to an existing notification channel (**Settings > Notification channels**), not a separate integration. Connect a Slack or Discord channel the normal way first, then turn on **Interactive approval buttons** on that channel's edit dialog. It is off by default, and the toggle requires the secret from the platform setup below, because without one there is nothing to verify an incoming click against.
## Set up the platform
Levelrail's Slack channel sends through an Incoming Webhook. To receive button clicks, that webhook must belong to a Slack app with **Interactivity** turned on.
Create one at [https://api.slack.com/apps](https://api.slack.com/apps), or use the app your Incoming Webhook already belongs to.
Under **Interactivity & Shortcuts**, turn Interactivity on and set the Request URL to:
```
https:///api/v1/webhooks/slack/interactions
```
Under **Basic Information**, copy the **Signing Secret** and paste it into the channel's interactive approval secret field. Levelrail uses it to verify that a click came from your Slack app.
Make sure the same app owns the Incoming Webhook URL the channel is configured with.
Discord's equivalent of a Request URL is an application's Interactions Endpoint URL, and verification uses the application's public key instead of a shared secret.
Open your application at [https://discord.com/developers/applications](https://discord.com/developers/applications) (the one your Incoming Webhook belongs to). Under **General Information**, copy the **Public Key** and paste it into the channel's interactive approval secret field. Save the channel with interactive approvals on first.
Set it to:
```
https:///api/v1/webhooks/discord/interactions
```
Discord sends a verification ping when you save and refuses the URL if the ping is not answered correctly, so the key must already be saved in Levelrail.
## What it does
* A deploy or promotion that needs approval posts a message with **Approve** and **Deny** buttons to every channel that has this turned on.
* A click decides the approval through the same code path as the dashboard's buttons: the same not-pending check, the same rule that the requester cannot approve their own request, and the same freeze gate. It cannot apply a change the dashboard would not allow.
* A forged or replayed request cannot approve anything. Every incoming request is signature-verified against the channel's own secret first, and an already-decided approval cannot be decided again (a second click or a retried delivery is told it is no longer pending).
**Identity scope**
Levelrail has no link between a Slack or Discord user and a Levelrail account. A verified click is authorized at the channel level: anyone who can click the button in that Slack channel or Discord server can decide the approval. The clicking user's name is recorded on the decision for visibility, but it does not authorize it. Treat the channel or server you point this at as the trust boundary.
## Endpoints
| Platform | Endpoint | Auth |
| - | - | - |
| Slack | `POST /api/v1/webhooks/slack/interactions` | `X-Slack-Signature` and `X-Slack-Request-Timestamp`, HMAC-SHA256 against the channel's signing secret |
| Discord | `POST /api/v1/webhooks/discord/interactions` | `X-Signature-Ed25519` and `X-Signature-Timestamp`, Ed25519 against the channel's public key |
Neither route uses a session or API token, like `POST /api/v1/webhooks/github/{name}`, because neither platform can present one. The signature check stands in for auth, and a request that fails it is rejected with no state change.
# CI
Source: https://docs.levelrail.com/ci
How the GitHub Actions checks on a pull request are chosen, what they cost, and which ones branch protection relies on.
How the GitHub Actions checks on a pull request are chosen and which ones gate a merge. This page is for contributors to Levelrail itself.
## The short version
A PR runs only the work its diff can affect:
* Docs only (`*.md`, `adr/`): change detection, the aggregator, the
secret scan and PR hygiene. A change under `docs/` also rebuilds the web
bundle, since `docs/` feeds the dashboard's `/help` pages.
* Web only (`web/`): the web job, with vitest limited to tests whose
module graph reaches a changed file. No Go job runs.
* Go: build, vet and tests for the affected packages only (the packages
you changed plus every package that imports them, including through test
imports), lint for the packages you changed, reporting only issues on
lines you touched.
* Workflow files only: actionlint on the changed workflow files, inside
the Lint job.
A store migration that only adds tables, indexes or columns doesn't run
everything. It selects the store and the packages that mention the table.
Whole-tree scanner tests have no import edge to what they scan, so
`scripts/ci-guards.txt` lists them and any non-test Go change selects them.
A `main` push runs the same impact-based jobs as a PR (`Build, vet`, `Lint`,
every `Test (*)` job, `Coverage gate`, `Web (tsc, eslint)`, `install.sh`),
scoped against `github.event.before` and falling back to `--full` when that
SHA is missing or unresolvable. The merge queue is gone, so this is the only
post-merge verification of the merged tree. Docs-only pushes never trigger
`ci.yml` at all (`paths-ignore` on the `push` trigger: `**/*.md`, `adr/**`,
`LICENSE`), and an area with no changed files skips as on a PR.
`nightly.yml` runs the full no `-short`, shuffled sweep once a day, as the safety
net for anything the scoping under-selects. It diffs against the last
green nightly, so a lane that failed or timed out is retried until it
passes. `-race` runs every night on the packages that start goroutines or use
`sync`, `atomic` or channels (`scripts/go-test-groups.sh race`), and on the api
and docker lanes on Sundays or a manual dispatch. The `rest` packages split on
`scripts/ci-heavy-packages.txt` so no lane nears its timeout, and the docker
lane pre-pulls its upstream images with retries.
## How a change is scoped
Two scripts decide, and you can run both locally against any diff.
`scripts/ci-changes.sh [head]` (the Detect changed areas job, no
Go toolchain) sorts changed files into areas:
| Changed path | Effect |
| - | - |
| `.github/workflows/ci.yml`, `scripts/ci-*.sh`, `scripts/affected-go-packages.sh`, `scripts/go-test-groups.sh`, the coverage, lane and migration check scripts | Full run of everything: the pipeline itself changed |
| `go.mod`, `go.sum` | Every Go package, full lint (same rule as the pre-push hook) |
| `internal/store/migrations/*.sql`, additive only (new table, index or column) | `internal/store` plus the packages whose Go code names the touched table, own tests only |
| `internal/store/migrations/*.sql` that drops, renames, updates, deletes, creates a view, or adds a trigger on an existing table | Every Go package (it can change behavior anywhere) |
| Any non-test `*.go` | Also the guard packages in `scripts/ci-guards.txt` (whole-tree scanners such as `test/execgate`, seconds each) |
| `.golangci.yml` | Full lint, plus the kit lane |
| `kit/*.go`, `kit/go.mod`, `kit/go.sum` | The kit lane, plus the root packages that import the changed kit package (`kit/go.mod` and `kit/go.sum` run every Go package) |
| `*.go` | That package, plus its dependents; lint for that package |
| Non-Go file inside a Go package (embedded schema, SQL, `testdata/`) | That package |
| File outside any package (`docs/`, root files, `proto/`) | Packages whose Go files name it in a string literal, for example `internal/api` for `docs/api-reference.md` |
| `web/` | Web job; config, lockfile or test setup changes run the full vitest suite |
| `docs/*.md` (except `docs/changelog/`, which the in-app help does not bundle) | Web bundle build and the help page tests |
| `scripts/test-ci-*.sh`, `scripts/ci-test-weights.txt`, `scripts/refresh-test-weights.sh` | The planner table tests (`Build, vet`), no Go test lane |
| `.github/actions/setup-go-cached/` | Every Go lane (live suites in smoke mode) |
| `install.sh`, `scripts/test-install-sh.sh`, `packaging/` | install.sh end to end on Ubuntu and Debian |
| Other `.github/workflows/*` | actionlint on the changed files |
| `.github/flaky-tests.txt` | The packages named in the changed lines, plus the quarantine lane |
| Anything else (`adr/`, `LICENSE`, other `.github/` files) | Nothing beyond the always-on checks |
`scripts/ci-go-plan.sh [head]` (inside the Plan Go work job) turns the Go
side into packages and test lanes through `scripts/affected-go-packages.sh`,
the same dependency-graph walk the pre-push hook uses. A change only a
package's own tests can see (`_test.go`, `testdata/`, a doc a test reads)
selects that package alone, not its dependents. Lanes:
* `internal/api` in four shards by test name whenever it is affected. It is
the slowest package by far, and most backend changes reach it. Tests are
packed onto shards by `scripts/ci-shard-names.sh`, heaviest first onto the
least loaded shard, using the seconds in `scripts/ci-test-weights.txt`
(unlisted tests weigh 0.28 s). Refresh the file from recent
`test-results-api-*` artifacts with `scripts/refresh-test-weights.sh`.
* `test/e2e` always gets its own `e2e` lane when it is affected, so the
heaviest package sets the floor alone. On a full live run
(see [Live suites](#live-suites)) the template fleet test gets its own
`e2e-fleet` lane, `e2e` runs everything else in that package, and
`test/e2e/reconcile` gets `e2e-reconcile` and `e2e-reconcile-2`: the three
catalog batch tests (about 200 s of the package's 430 s) run in the second
lane, every other test, including any new one, in the first.
* Up to `CI_SMALL_LANE_MAX` (default 8) other affected packages share one
`rest` lane. Above that, Docker-backed packages get their own `docker`
lane with bounded `-p` and the rest share `rest`.
* No lane at all when nothing Go-related is affected.
The lanes and the checks that gate them:
| Lane | Packages | Present when | Required check name |
| - | - | - | - |
| `api-1` to `api-4` | `internal/api`, sharded by test name | `internal/api` affected | `Test (api-N)`, aggregated as `Test (internal/api)` |
| `e2e-fleet` | `test/e2e`, template fleet test only | `test/e2e` affected and a fleet path changed ([below](#which-e2e-lanes-run)) | `Test (e2e-fleet)` |
| `e2e` | `test/e2e` (minus the fleet test on a full live run) | `test/e2e` affected and an e2e path changed | `Test (e2e)` |
| `e2e-reconcile` | `test/e2e/reconcile`, minus the three catalog batch tests on a full live run | package affected and a fleet path changed | `Test (e2e-reconcile)` |
| `e2e-reconcile-2` | the three catalog batch tests of `test/e2e/reconcile` | as above, full live run only (smoke mode runs one lane) | `Test (e2e-reconcile-2)` |
| `docker` | other Docker-backed packages, `-p 2` | many packages affected | `Test (docker)` |
| `rest` | everything else (and the Docker ones on a small diff) | any other package affected | `Test (rest)` |
Every planned lane is listed in `rest_checks` or `api_checks`, which the
`Test (everything else)` and `Test (internal/api)` aggregators wait on, and
`scripts/test-ci-go-plan.sh` asserts that for each case, so a lane cannot be
added without a check that fails when it fails or goes missing. Planning runs
in its own Plan Go work job, so the test lanes start as soon as the plan
exists instead of waiting behind `go build` and `go vet`.
`test/e2e` is split from `test/e2e/reconcile` along one dependency: only the
former constructs a live `api.Router`. Since `internal/api` is the package
most PRs touch, this keeps a one-file fix there from selecting the whole
end-to-end suite through the test-import walk. The reconciler and Docker-only
tests in `test/e2e/reconcile` run only when a PR's diff reaches something they
import.
## Which e2e lanes run
The three live-Docker lanes are the most expensive jobs on a PR (each about
6 to 7 minutes of runner time, often all three at once). The plan only
schedules a lane when the diff can reach what it exercises. A lane that is
not planned is not in `rest_checks`, so `Test (everything else)` does not wait
on it and `CI required` passes without it.
| Lane | Runs when the diff touches |
| - | - |
| `e2e-fleet`, `e2e-reconcile` | `internal/reconcile/` (a diff confined to `internal/reconcile/ingress/` does not count), `internal/agent/`, `internal/network/`, `test/e2e/` |
| `e2e` | `internal/ingress/`, `internal/reconcile/` (including `ingress/`), `internal/docker/`, `internal/deploy/`, `test/e2e/` |
| all three | The `ci:live` label, a full run, or a change to `ci.yml`, `nightly.yml`, `.github/actions/` or the planner, lane runner or group scripts (alongside a diff that reaches `test/e2e`) |
`nightly.yml` is unchanged and runs every lane and the full live set, so a
change this table skips is still verified within a day. Add `ci:live` to a PR
that changes something the table does not list but that a template deploy
depends on (for example `internal/catalog`).
`scripts/test-ci-go-plan.sh` holds the table cases: an ingress-only diff
plans `e2e` and neither `e2e-fleet` nor `e2e-reconcile`, a reconcile diff
plans all three, a docs-only diff plans none, and `ci:live` plans all.
## Live suites
The heaviest tests pull real upstream images and run real container
lifecycles: the template fleet (`TestTemplateFleet_Live_DeploysAndTearsDownCleanly`,
about 1,000 s when run one template at a time), the catalog batch live tests
and the break-glass tests. They are the only tests the PR gate trims. Every
other `test/e2e` and `test/e2e/reconcile` test, and every unit test, runs on
every PR the same as before.
`scripts/ci-go-plan.sh` decides `live=full` or `live=smoke` for the PR, in one
place, and prints the reason in the Go plan job summary:
| `live` | When | What runs |
| - | - | - |
| `full` | The diff touches `internal/catalog`, `internal/compose`, `internal/registrycatalog`, `internal/reconcile`, `internal/docker`, `internal/dockertest`, `internal/agent`, `internal/network`, `internal/build`, `internal/pipeline`, `internal/ingress`, `test/e2e`, `ci.yml`, `nightly.yml`, `.github/actions`, or the planner, lane runner or group scripts | The whole fleet (28 templates) when `e2e-fleet` is planned, every catalog batch, both break-glass tests |
| `full` | The PR has the `ci:live` label | Same |
| `full` | A full run (`go.mod`, migrations, pipeline scripts, push with no base) | Same |
| `smoke` | Anything else | Fleet limited to `uptime-kuma`, `mailpit` and `gitea` (one single-service, one tiny, one db-backed multi-service template); the catalog batch and break-glass tests skip |
A skipped heavy test is never silent. It skips with the message
`skipped by design: LEVELRAIL_LIVE_SUITE=smoke`, and the lane's job summary
lists every test skipped that way under "Skipped by design". The mechanism is
`LEVELRAIL_LIVE_SUITE` in `test/e2e/testenv`; unset (local runs and
`nightly.yml`) means everything runs.
To force the full set on a PR: add the `ci:live` label (create it once with
`gh label create ci:live`), then re-run all jobs or push. The plan job reads
the labels live, because a re-run replays the original event payload.
`nightly.yml` always keeps the docker lane in its matrix
(`NIGHTLY_LIVE_ALWAYS`), so the full live set runs every night even on a day
with no commits, which is when an upstream image change would otherwise go
unnoticed.
Run the full set on a developer machine with one command:
```sh theme={"dark"}
scripts/live-tests.sh # all of test/e2e and test/e2e/reconcile
scripts/live-tests.sh -run TestBreakGlass # one suite
```
The fleet test deploys `LEVELRAIL_FLEET_PARALLEL` templates at once (default
3, `1` for strictly sequential). Each template gets its own app id, network
and volumes, and teardown asserts nothing leaked per template.
The coverage gate follows the plan: a full run checks the 70% aggregate
for `internal/`, a scoped run checks the changed-line gate only, at a
lower 50% bar (a partial profile makes the aggregate meaningless, same as the
pre-push hook).
## The kit module
`kit/` is a nested Go module (`github.com/GLINCKER/levelrail/kit`), so
`go test ./...`, `go vet ./...` and `golangci-lint run` from the repo root
never reach it. It has its own lane instead:
* The Kit (vet, lint, test) lane runs only when a diff touches `kit/` (not `kit/*.md`),
`.golangci.yml`, or the pipeline itself. It runs `go -C kit vet`, golangci-lint
with the root `.golangci.yml` (found by walking up from `kit/`),
`go -C kit test -short` and `scripts/check-brand-strings.sh`.
* Coverage: a 70% floor for all of `kit/` on every run of the lane, and the 50%
changed-line bar from the main gate, both through the same coverage scripts
with the `kit/` prefix.
* Dependents: `scripts/affected-go-packages.sh` maps a changed kit package to its
import path and walks the root import graph, so root packages that import it are
tested in the normal lanes. Test-only kit changes select no root package.
* The lane is a `needs` of `CI required`, so a skipped lane (no kit change) passes
and a failed or cancelled one blocks.
* `nightly.yml` runs `go -C kit test -race -shuffle=on` without `-short`.
* Releases are independent: release-please tags the module `kit/vX.Y.Z`, which does
not match `release.yml`'s `v*` tag filter, so it never starts a product release.
The kit package carries `release-as: 0.1.0` until its first release ships; remove
it afterwards.
## Required checks
`CI required` is the aggregator and the one check meant to gate a merge. It always runs and needs `changes`, `plan`, `build-vet`, `lint`, `kit`, the Go test jobs, `coverage-gate`, `web-check` and `install-sh`. It fails if change detection did not succeed or any job it needs failed or was cancelled, and a job that was skipped because its area had no work counts as passing. Because branch protection can name just this one context, the job set can be split or renamed without touching the ruleset. Branch protection itself is configured in GitHub, not in this repository.
Seven jobs also never skip because of an upstream failure: `Plan Go work`, `Build, vet`, `Lint`, `Test (internal/api)`, `Test (everything else)`, `Coverage gate` and `Web (tsc, eslint)`. Each runs, and fails, whenever an upstream job failed or was cancelled, and is skipped only when its area has no work for the diff. GitHub reports a job skipped by its own `if` as success for a required check, so auto-merge does not wait on it.
## Other workflows on a PR
| Workflow | On a PR | Why |
| - | - | - |
| `secret-scan.yml` (gitleaks) | Every PR, commits in the change only | A token leaked in a markdown file is still a leak. Uses the pinned release binary, no Go build |
| `pr-hygiene.yml` | Every PR | Labels, size, description and commit rules in one job |
| `codeql.yml` | No | Not a required check. Runs on pushes to `main` that touch Go or web sources, weekly, and on manual dispatch |
| `dependabot-auto-merge.yml` | Dependabot PRs only (listed as skipped elsewhere) | |
| SonarCloud | Every PR | A GitHub App, not an Actions job: it runs on the vendor's infrastructure, uses no Actions minutes, and is not a merge blocker |
## Scheduled, not PR-triggered
* `branch-cleanup.yml`: deletes a merged PR's head branch immediately, plus a
weekly sweep (Monday) for anything left over from before branch deletion
on merge was enabled.
* `backup-tags.yml`: moves `backup/daily` to `main`'s tip every day, and
`backup/weekly` on Mondays, as a known-name emergency rollback target.
Tags, not branches, so they never show up in the list the cleanup above
is shrinking. Every commit on `main` is already a valid, permanent
rollback point on its own (no force-push, no deletion); these tags exist
only so finding one doesn't mean hunting for a SHA by hand first.
## Versioning and releases
Versions are automatic. [release-please](https://github.com/googleapis/release-please) reads the conventional commit messages on `main`, picks the next version, updates `CHANGELOG.md`, and keeps one open pull request titled `chore(main): release `. Merging that PR tags the release, creates the GitHub Release, and dispatches `release.yml`, which builds every binary and publishes to Homebrew and npm.
The one manual step is merging that PR. By default release-please uses the built-in `GITHUB_TOKEN`, and GitHub does not run CI on pull requests opened that way, so the required `CI required` check never reports and the PR stays blocked until someone reopens it or merges around the gate.
To make releases hands-off, give release-please a real token. A fine-grained personal access token limited to this repository, with Contents and Pull requests set to read and write, is enough:
```bash theme={"dark"}
gh secret set RELEASE_PLEASE_TOKEN --repo glincker/levelrail # paste the token at the prompt
gh variable set AUTO_RELEASE --body true --repo glincker/levelrail
```
With the secret, the release PR runs CI like any other PR, and the tag push starts `release.yml` by itself. With the variable as well, release-please also arms auto-merge on the PR, so a change on `main` ships as a new beta once CI passes. Every merge to `main` updates the same PR, so this releases on every merge. Leave `AUTO_RELEASE` unset to keep merging release PRs by hand, with the token still doing its job of running CI on them.
## Publishing the CLI to Homebrew and npm
The release workflow builds every binary, attaches them to the GitHub Release, and then runs two independent jobs. A failure in one never blocks the other or the release itself.
### Homebrew
`publish-homebrew` renders `Formula/levelrail-cli.rb` from the release's `checksums.txt` and pushes it to `glincker/homebrew-tap` over SSH. It uses a deploy key that has write access to that one repo (the `HOMEBREW_TAP_DEPLOY_KEY` secret), and the step skips when the secret is missing. Users install with `brew install glincker/tap/levelrail-cli`. A personal tap has no popularity requirement; only `homebrew/core` does.
One-time setup:
```bash theme={"dark"}
ssh-keygen -t ed25519 -N "" -C levelrail-release-tap -f ./tapkey
gh repo deploy-key add ./tapkey.pub --repo glincker/homebrew-tap --title "levelrail release" --allow-write
gh secret set HOMEBREW_TAP_DEPLOY_KEY --repo glincker/levelrail < ./tapkey
rm ./tapkey ./tapkey.pub
```
The formula template is `packaging/homebrew/levelrail-cli.rb.tmpl`, and `scripts/test-packaging.sh` checks the rendered result. The binary is a raw download without the execute bit, so the template sets it with `chmod`; without that, `brew install` fails with `EACCES`.
### npm
`publish-npm` publishes `levelrail-cli` plus six platform packages (`levelrail-cli--`) using npm trusted publishing, so no npm token is involved. Stable releases take the `latest` dist-tag and prereleases use `beta`. A version already on the registry is skipped, so re-running a partly failed release finishes the rest. The job only runs when the `NPM_PUBLISH_ENABLED` repo variable is `true`.
One-time setup. npm configures a trusted publisher on a package that already exists, so each package is published once by hand first. The script does the whole thing under your own npm login:
```bash theme={"dark"}
scripts/npm-bootstrap.sh --dry-run # builds the packages and shows every step, changes nothing
scripts/npm-bootstrap.sh # publishes, registers the trusted publisher, then enables the job
```
It downloads the newest release's binaries, builds the seven packages, publishes the platform packages first, runs `npm trust github` on each package for `glincker/levelrail`, workflow `release.yml`, environment `npm-publish`, and only then sets `NPM_PUBLISH_ENABLED`. It skips anything already done, so it is safe to re-run after a failure. Publishing and registering both ask for your npm two-factor code.
After that, every release publishes with no token. If a publish fails, use "Re-run failed jobs" on the workflow run; versions already on the registry are skipped.
An `NPM_TOKEN` secret, including one set at the organization level, is not used and can stay as it is for other repositories.
## Caching
Set the repository variable `GOCACHE_BUCKET` (plus `GOCACHE_REGION`,
`GOCACHE_ENDPOINT` for R2, Tigris or MinIO, and the secrets `GOCACHE_KEY_ID`
and `GOCACHE_SECRET`) to switch every Go job to a shared remote build and test
cache through `GOCACHEPROG` ([gobuildcache](https://pkg.go.dev/github.com/richardartoul/gobuildcache)).
A package whose inputs hash the same is a cache hit, build and test result
alike, across PRs, nightly and laptops. Unset, the actions/cache path below
applies. Live Docker lanes should keep `-count=1`.
* Go: `~/.cache/go-build` and `~/go/pkg/mod`, one cache per job kind (build,
lint, and each test lane group), keyed on the Go version and `go.sum` plus
`tools/go.sum`. Pushes to `main` save one fresh cache per day, in the job's
post step so it holds what the job compiled. All test lanes share one cache
(`go-test-all`, saved by the first lane). PR runs only
restore, so they cannot churn the repository's 10 GB cache quota. Every
job that needs this shares `.github/actions/setup-go-cached` instead of
repeating setup-go plus restore/save inline.
* golangci-lint: the action's own analysis cache, saved on `main` only.
* npm: `actions/setup-node`'s npm cache, keyed on `web/package-lock.json`.
* Dependabot: its PRs go through the same scoping. A GitHub Actions bump touches
only `.github/workflows/*`, so it runs actionlint and nothing else; an npm bump
runs the web jobs only; a `tools/` bump runs the release notes generator; a
`go.mod`, `go.sum` or `kit/go.mod` bump runs every Go lane, because the import
graph cannot scope a dependency change.
* Release image cache: `release.yml` writes its BuildKit layers with
`cache-to: type=gha`, and a tag run can only read caches of its own tag, so
each release leaves about 5 GB that no later run reads and that competes with
the CI caches for the 10 GB quota.
* tsc: `web/node_modules/.tmp` (the `tsc -b` build info), restored on PRs and
saved on `main` pushes. A stale build info is safe: tsc compares file hashes.
## Web jobs
`Web (tsc, eslint)` builds, checks the bundle budget, lints changed files and
runs the related vitest tests. When the whole vitest suite is selected
(config, lockfile or test-setup changes), it runs separately as
`Web tests (1/4)` to `Web tests (4/4)` with `vitest --shard`, because vitest
runs files serially (`fileParallelism: false`, to avoid jsdom contention
flakes) and one job would be the suite's whole wall time. A skipped shard job
passes `CI required` the same way a skipped area does.
## Concurrency and merge queue
A new push to a PR cancels that PR's previous CI run (`ci.yml`, `secret-scan.yml`,
`pr-hygiene.yml`). The group key includes the event name and, outside pull
requests, the commit SHA, so `merge_group` and `main` runs never share a
group with a PR and are never cancelled; `nightly.yml` has no group.
There is no merge queue on `main` now, so a PR merges as soon as its own
`CI required` check passes, and the `push` run on `main` is the only
verification of the merged tree. That run does everything a PR run does,
scoped by the same impact detection, so a change that was fine alone but
breaks against a newer `main` is caught the same day, not at the nightly
sweep.
`ci.yml` still listens for `merge_group`, with the same base-commit scoping
as a PR, so re-enabling a queue needs only the branch protection setting.
If a queue returns, the `push` run would duplicate the `merge_group` run on
the same tree, so put a `github.event_name != 'push'` guard back on the heavy
jobs at that point.
`Build, vet` also refreshes the daily Go build cache on `push`
(the cache `save` flag, true only for push events), which PR runs depend on
staying warm. `codeql.yml` and `secret-scan.yml` are separate workflow
files with their own `push` triggers, unaffected by anything in `ci.yml`.
## Check what a diff would run
Both scoping scripts run locally against any pair of commits, which is the quickest way to see what a change selects.
`scripts/test-ci-changes.sh` holds the table cases for `ci-changes.sh` (docs-only,
web-only, lockfile-only, script-only, Go-only, `go.mod`, workflow, planner and
embedded-file changes), the shard partition checks and a fake-`gh` check that a
skipped, missing or failed lane fails the aggregator that `CI required` waits on:
```sh theme={"dark"}
scripts/ci-changes.sh
scripts/ci-go-plan.sh
```
## Measured timings
Baseline: the last 60 `ci.yml` runs (48 of them on pull requests). Median
wall time per job, from the run and step timestamps.
| Job | Median | p90 |
| - | - | - |
| Whole PR run | 491 s | 622 s |
| `Test (api-1..4)` (internal/api shards) | about 390 s | about 450 s |
| `Test (e2e)` / `e2e-fleet` / `e2e-reconcile` | 382 s / 468 s / 475 s | 412 s / 490 s / 525 s |
| `Test (rest)` / `Test (docker)` | 223 s / 182 s | 283 s / 217 s |
| `Build, vet` (planner table tests about 128 s of it) | 221 s | 266 s |
| `Lint` | 183 s | 237 s |
| `Web (tsc, eslint)` (`vitest related` has rare outliers) | 152 s | 375 s |
| `Plan Go work`, `Coverage gate`, `CI required` | 49 s, 12 s, 3 s | |
The first job starts 2 to 57 s after the run is created. Go build and
module caches hit on PR runs. The tsc build-info cache never hits, because
`tsc -b` re-checks the whole project on every run (about 31 s) and
`web/tsconfig.app.json` sets no `incremental`. Enabling it is a tsconfig change
that needs maintainer approval; the cache save step would then need to allow
PR runs as well.
Local hooks, one small sample diff each on a warm cache (seconds):
| Stage | Before | After |
| - | - | - |
| pre-commit, Go file only | 2.3 | 0.7 (warm cache both runs, so not a gain of this change) |
| pre-commit, one web file | 35.3 | 29.5 (eslint and prettier now run beside `tsc -b`) |
| pre-push, Go file (`internal/brand`) | 59.4 | 33.8 |
| pre-push, web file only | 21.0 | 1.7 (`go build ./...` skipped when nothing Go-affecting changed) |
| internal/api narrowing (`affected-api-tests.sh`) | 36 | 13 |
A fresh worktree has no `web/src/routeTree.gen.ts` (gitignored, written by
`vite build`), so pre-commit used to fail on every web change there. It now
runs `vite build` once when the file is missing, about 20 s.
pre-push bounds every `go test` it runs with `-timeout`, default `30m`,
overridable with `LEVELRAIL_PUSH_TEST_TIMEOUT`. The default `go test`
timeout of 10 minutes is shorter than a whole internal/api run.
## Expected effect of the lane changes
Estimates from the artifacts of three recent runs, to be compared with the table
above after the first runs on the new layout.
| Change | Expected | Basis |
| - | - | - |
| Weighted api shards | api lanes lose about 25 s of test time (the slowest shard's summed test time 303 s to 276 s; shards were 232 to 303 s) | per-test means of 3 runs, packed greedily |
| `e2e-reconcile` split in two | the reconcile lane drops from about 480 s to about 280 s; with `e2e-fleet` (about 410 s) now the longest e2e lane, a run that plans the fleet lanes saves roughly 65 s of wall time | per-test times of the reconcile package |
| Planner tests on planner-only changes | no wall effect; a change to the planner tests alone no longer skips them | `ci-changes.sh` planner output |
The api shard balance is noisy run to run (a shard's summed test time varied by
up to 30% between runs with the same assignment), so the gain from weights is
smaller than the spread.
## Runners
Every job reads `runs-on` from the `CI_RUNNER` repository variable and falls
back to `ubuntu-latest`, so trying a faster provider (Blacksmith, Namespace,
RunsOn) is a variable change, not a workflow edit.
## Blast radius
`scripts/check-fanin.sh` prints how many packages transitively depend on each
package and fails when one passes its budget in `scripts/fanin-budget.txt`.
Today `internal/store` reaches \~60 packages through `internal/spec`, so most
store edits select most of the tree. Lowering those numbers (consumer-side
interfaces, moving shared types to leaf packages) is the lasting fix for
"one edit runs 1000 tests".
## Tuning
* `CI_SMALL_LANE_MAX`: the package count at or under which non-api
packages share one lane (default 8), read by `scripts/ci-go-plan.sh`.
* To force the full suite on a PR, change `.github/workflows/ci.yml` or one
of the pipeline scripts, or run the nightly workflow on the branch with
`workflow_dispatch`.
* The e2e and docker lanes' `go test -timeout` (27m, job `timeout-minutes: 32`)
has headroom above the fleet test's cost. The fleet test now runs its
templates in bounded parallel (`LEVELRAIL_FLEET_PARALLEL`), and the PR
smoke subset keeps the typical `test/e2e` lane in the low minutes.
* Test databases: `internal/store`, `internal/pipeline` and `internal/api`
tests start from a copy of one migrated template database instead of
replaying every migration per test. New test helpers that open a store in
a hot path should do the same.
# Levelrail CLI Reference
Source: https://docs.levelrail.com/cli-reference
Complete reference of all Levelrail CLI commands, organized by group with examples and typical use cases.
`levelrail-cli` is a scriptable client for the control plane's HTTP API. Everything it does goes through `/api/v1`, with the same tokens and permissions as the dashboard, so you need no SSH key and no open port on your servers. This page lists every command group with its subcommands and the flags you will use most. Run `levelrail-cli` with no arguments for the command list, or `levelrail-cli -h` for any command's own flags.
Commands are shown as `levelrail-cli ...`. If you rename the binary, the command name follows it (the CLI reads its name from `os.Args[0]`). The server is a separate binary, `levelrail`, which has a few maintenance commands of its own, such as `setup-token`, `restore-snapshot`, `restore-db`, `recover-admin`, `upgrade-note` (record who swapped the binary by hand), and `healthcheck`. Those are in [Installing](/installing) and [Disaster recovery](/disaster-recovery).
## Install and sign in
`curl -fsSL https://levelrail.com/install-cli.sh | sh`
```bash theme={"dark"}
export APP_API_URL=https://console.example.com
levelrail-cli auth login --device
levelrail-cli auth whoami
```
`--device` prints a short code that you approve in the dashboard, the same model as `gh auth login`. Credentials are saved to `~/.config/levelrail-cli/credentials`. For CI, create an API token under **Settings, API tokens** and pass it with `--token` or `APP_API_TOKEN`.
Each command resolves its target in this order, and an explicit flag always wins:
| Setting | Flag | Environment variable | Default |
| - | - | - | - |
| API token | `--token` | `APP_API_TOKEN` | the saved profile |
| Control plane URL | `--api-url` | `APP_API_URL` | `http://localhost:8080` |
| Profile | `--profile` | `APP_PROFILE` | `default` |
Manage more than one control plane with named profiles: `levelrail-cli auth login --profile staging`, then `--profile staging` on any command. `levelrail-cli profile list` shows them.
## Scripting: `--json` and exit codes
Every command that returns data or a result supports `--json` (shorthand for `--output json`), and `--query` takes a JMESPath expression. With `--json`, stdout carries only the JSON result. The exceptions are `completion bash|zsh|fish` (a shell script) and `control-plane-backups help-dr` (a static runbook). A test walks the command tree and fails when a new command has no `--json` and is not on that exempt list.
On failure, `--json` also prints an error object to stdout (the message still goes to stderr):
```json theme={"dark"}
{"error": "server returned 404: app not found", "code": "not_found", "exit_code": 4, "http_status": 404, "hint": "check the resource name with the matching list command"}
```
`error` is the original field. `code` is one of `validation`, `network`, `unauthorized`, `forbidden`, `not_found`, `conflict`, `rate_limited`, `invalid_request`, `server_error`, `api_error`. `http_status`, `retry_after` (rate limits) and `hint` appear when they apply.
Exit codes are stable and shared by every command:
| Code | Meaning |
| - | - |
| 0 | Success |
| 1 | Usage error (unknown command, missing argument, bad flag); also a failed check for `control-plane-backups verify` and a critical item for `attention` |
| 2 | Validation error: well-formed flags, but the request they describe is invalid |
| 3 | Network error: the control plane could not be reached |
| 4 | API error: the control plane replied with a non-2xx status (use `code` and `http_status` in the JSON error to tell auth, not found and conflict apart) |
| 5 | Deploy failed: `apps wait` reached a failed deploy |
| 6 | Deploy timeout: `apps wait` or `apps deploys wait` gave up before the deploy converged |
| 7 | `apps deploys wait` ended on a deploy that failed, was canceled, superseded or blocked |
Codes 3 and 4 are broad on purpose so existing scripts keep working; the JSON error object carries the finer distinction.
## Scripting: `--debug`
`--debug` works anywhere on the command line (before or after the subcommand) and traces every outgoing request this invocation makes to stderr: method, URL, and the response status and timing, one line per call. It never prints request or response headers or bodies, so `Authorization` and any token value never appear, even in debug mode; a token that somehow ended up in a URL's query string is also redacted. stdout is untouched, so `--debug` composes with `--json`/`--query` for scripting.
```
levelrail-cli apps list --debug --api-url http://10.0.0.5:8080
```
```
DEBUG: GET http://10.0.0.5:8080/api/v1/apps -> 200 OK (42ms)
```
## Experimental command groups
Some groups are off by default and stay hidden from `--help` and shell completion until the shell you run the CLI from sets `APP_EXPERIMENTAL`. This page marks them. The keys are `ai-chat` (`ai`, `settings ai-assistant`), `ai-models` (`models`), `load-balancer` (`lb`), `iac` (`apply`, `diff`, `export`) and `cloudflare-tunnel` (`cloudflare-tunnel`). See [Experimental features](/experimental-features).
## Web CLI console: host diagnostics
The dashboard's CLI console (experimental key `cli-console`) runs the commands on this page against the control plane. It also answers a small `host` namespace itself, inside the control plane process, without starting the CLI or any other program. It is read only and is not a shell.
| Command | Shows |
| - | - |
| `host df` | Disk usage of the data directory and `/` |
| `host free` | Memory total, used, available, and swap |
| `host uptime` | Time since the server booted |
| `host load` | 1, 5 and 15 minute load averages and the CPU count |
| `host os` | OS name (`PRETTY_NAME` from os-release), kernel release and architecture |
| `host service` | The control plane's version, start time, uptime, goroutines, memory (RSS) and whether a restart supervisor (systemd, Docker) was detected |
| `host help` | This list |
None of them take arguments: paths, process names, container names and flags are refused. Each line goes through the same step-up, one-command-at-a-time, idle timeout, output cap and audit log (`cli_console.run`) as any other console command. Linux only; other platforms reply "not supported on this platform". Set `APP_CLI_CONSOLE_HOST=off` on the server to turn the namespace off while keeping the rest of the console.
## Top-level convenience aliases
```
levelrail-cli deploy --image IMAGE [flags]
```
Alias for `apps deploy`; deploy an image to an existing app.
```
levelrail-cli rollback --image IMAGE [flags]
```
Alias for `apps rollback`; redeploy an older image.
## Apps
`apps list` prints every app the caller can read as a bare JSON array. On a fleet large enough that matters, add `--max-items N` to cap the page size; the response becomes `{"items": [...], "next_token": "...", "total_count": N}` instead, and an empty `next_token` means there is no further page. Pass that `next_token` back in as `--starting-token` to fetch the next page (requires `--max-items`, since the server only pages a result when a limit is set):
```
levelrail-cli apps list --max-items 50 --json
levelrail-cli apps list --max-items 50 --starting-token 50 --json
```
Run `levelrail-cli apps -h` for the full flag list of any command below. Deploy lifecycle is covered in [Deploying and managing apps](/deploying-apps).
### Create, inspect and list
```
levelrail-cli apps create --name NAME --image IMAGE --port PORT [flags]
levelrail-cli apps create --name NAME --port PORT --repo URL --image-repo REPO [flags]
levelrail-cli apps create --file app.yaml [flags]
levelrail-cli apps create --interactive
levelrail-cli apps list [--max-items N] [--starting-token T] [flags]
levelrail-cli apps get [flags]
levelrail-cli apps status [flags]
levelrail-cli apps overview [name ...] [flags]
levelrail-cli apps network [flags]
levelrail-cli apps validate --file [flags]
levelrail-cli apps clone [flags]
levelrail-cli apps delete [flags]
```
* `apps create` registers an existing image or triggers a real build from a git repository; `--file` reads an [app spec](/app-spec-reference), `--interactive` runs a step-by-step wizard. Required flags missing outside `--interactive` fail with an error instead of prompting.
* `apps status` shows an app's current reconcile conditions; `apps overview` shows CPU, memory, traffic and errors for every app (or the names given) in one request; `apps network` shows the live traffic path (container port, host port, running).
* `apps validate` parses and validates an app.yaml or a Docker Compose file locally, with no API call and no deploy. It prints the detected format, service count, and every non-blocking `notices` entry a real deploy would also surface.
* `apps delete` removes an app and stops its containers; if the node is unreachable the delete is reported as pending and retried.
### Deploy, roll back and scale
```
levelrail-cli apps deploy --image IMAGE [flags]
levelrail-cli apps deploy-compose --file compose.yaml [flags]
levelrail-cli apps deploy-spec --file app.yaml --repo-url --ref [ [flags]
levelrail-cli apps builds trigger --repo URL --ref REF [flags]
levelrail-cli apps rollback --image IMAGE [flags]
levelrail-cli apps promote --to ENVIRONMENT_ID [--target NAME] [--preview] [flags]
levelrail-cli apps wait [--attempt-id ID] [--timeout 5m] [--poll-interval 3s] [flags]
levelrail-cli apps restart [flags]
levelrail-cli apps apply [flags]
levelrail-cli apps stop [flags]
levelrail-cli apps keep [flags]
levelrail-cli apps start [flags]
levelrail-cli apps scale --replicas N [--strategy rolling|recreate|blue-green] [flags]
levelrail-cli apps group [flags]
levelrail-cli apps hook-runs [flags]
levelrail-cli apps images [flags]
levelrail-cli apps set-node [--with-volumes] [flags]
levelrail-cli apps clear-node [--with-volumes] [flags]
levelrail-cli apps moves list [flags]
levelrail-cli apps moves get [flags]
```
* `apps keep` turns a one-click trial deployment into a permanent app: it stops being flagged as throwaway, with every service of the same app. Repeating it is harmless; an app that never was a trial answers 409.
* `apps wait` polls until a deploy attempt converges and exits 0 on success, 5 if it converged as a failure, 6 on timeout. On success it says what happened: `rolled out`, `already up to date` (the deploy changed nothing) or `restarted`; `--json` carries the same as `outcome`.
* `apps deploy-spec` fans an app.yaml's `services:` map out into independent builds under one app; `apps group` shows a service's siblings under the same multi-service app.
* `apps builds trigger` builds an image from a git source and deploys it to an existing app.
* `apps promote` promotes the app's image onto a sibling app in another environment.
* `apps apply` restarts an app so saved env, secret and config changes reach the running container; it does nothing when nothing is pending.
* `apps scale` changes the replica count and/or the deploy strategy.
* `apps hook-runs` shows the most recent outcome of an app's pre and post deploy hooks; `apps images` lists the locally present image tags under the app's current image repo.
* `apps set-node` and `apps clear-node` move an app to another node, or back to the control plane's own local node. With `--with-volumes` the named volumes move too, and `apps moves` shows the history of those moves.
### Deploy history, approvals and automation
```
levelrail-cli apps deploys list [flags]
levelrail-cli apps deploys show [deploy-id] [flags]
levelrail-cli apps deploys wait [deploy-id] [--timeout 10m] [--poll-interval 2s] [flags]
levelrail-cli apps deploys compare --from ID [--to ID] [flags]
levelrail-cli apps deploys logs [flags]
levelrail-cli apps deploys steps [flags]
levelrail-cli apps deploys failed [--since 24h] [flags]
levelrail-cli apps deploys cancel [flags]
levelrail-cli apps deploys rollback-to [--watch] [flags]
levelrail-cli apps deploys gate [deploy-id] [--watch] [flags]
levelrail-cli deploy status [deploy-id] [--watch] [flags]
levelrail-cli apps deploy-safety show|set|targets [flags]
levelrail-cli apps timeline [--limit N] [flags]
levelrail-cli apps diagnose [--deploy ID] [--apply-fix N] [flags]
levelrail-cli apps preflight [--require-env A,B] [flags]
levelrail-cli apps freeze show [flags]
levelrail-cli apps freeze set --cron EXPR --duration D [--timezone TZ] [--reason TEXT] [flags]
levelrail-cli apps freeze clear [flags]
levelrail-cli apps schedule set --cron EXPR --branch NAME [flags]
levelrail-cli apps schedule get [flags]
levelrail-cli apps schedule history [flags]
levelrail-cli apps cancel-superseded enable|disable|status [flags]
levelrail-cli apps auto-rollback enable|disable|status [flags]
levelrail-cli functions deploy|list|invoke|delete [flags]
levelrail-cli jobs list|get|create|update|run|history|logs|enable|disable|delete [] [flags]
levelrail-cli apps sleep enable|disable|status|wake|now|history [flags]
levelrail-cli apps sleep list [flags]
levelrail-cli apps canary start|status|weight|promote|abort [flags]
levelrail-cli apps auto-update enable|disable|status|check [flags]
levelrail-cli apps auto-rollback-slo-burn set off|auto|dry_run|pause_for_human [flags]
levelrail-cli apps auto-rollback-slo-burn status [flags]
```
* `apps deploys list` is real, row per attempt deploy history, newest first. `apps deploys show` prints one attempt (the newest by default) with its structured failure: code, cause, failing step, redacted log excerpt, suggested fix, docs link and retryable. See [Deploy failures](/deploy-failures).
* `apps deploys wait` blocks until one deploy is healthy, failed, canceled, superseded or blocked (a freeze window or approval) and prints the result with its failure. Exit 0 healthy, 7 failed, canceled, superseded or blocked, 6 timeout. Without a deploy id it resolves the newest deploy once and follows it by id.
* `apps deploys compare` diffs two deploy attempts, or one against the current live state. `apps deploys logs` prints one attempt's full build and log output to stdout. `apps deploys steps` streams the pipeline steps (detecting, building, pushing, deploying) until the attempt ends and exits non-zero if a step failed; an already finished attempt replays only a short summary.
* `apps deploys failed` lists every app's latest failed deploy in the window (default set by the server), with the image of its newest good deploy as a rollback target.
* `apps deploys cancel` cancels a queued or in-progress deploy before it cuts traffic. `apps deploys rollback-to` redeploys a past succeeded deploy's exact image, pinned by digest or rollback pin, with that deploy's config; `--watch` follows it through the readiness gate.
* `apps deploys gate` (alias `deploy status`) prints a release's gate timeline: readiness passes, stable window, cutover, drain, rollback pin and post-deploy watch. `--watch` follows it until the release clears its watch, fails, is rolled back or is replaced. Exit 0 healthy or in progress, 7 failed or rolled back. `apps deploy-safety` shows and overrides the per-app thresholds (`set web --stable-window 20s --auto-rollback false`, `default` clears one) and lists pinned rollback targets. See [Deploy safety](/deploy-safety#readiness-gate).
* `apps timeline` shows what happened to an app, newest first: deploys, rollbacks, restarts, env, secret and config changes (key names only, never values), scaling, stop and start.
* `apps diagnose` explains a failed deploy or crashloop and can apply a suggested fix with `--apply-fix N`. `apps preflight` runs pre-deploy checks (DNS, ports, disk, image, env).
* `apps freeze` manages deploy freeze windows: a window starts at every match of a 5 field cron expression, evaluated in `--timezone` (default UTC), and lasts `--duration`. While one is active, webhook, pipeline and released deploys are held and run when the window ends; a manual deploy needs `apps deploy --override-freeze --override-reason TEXT`. `set` replaces the app's windows with the one given. Fleet-wide windows are under `settings deploy-freeze`.
* `apps schedule` configures a recurring redeploy of a branch's latest commit on a cron schedule.
* `apps cancel-superseded` lets a newer queued deploy replace older queued ones of the same branch. Only queued deploys are replaced; a deploy that already started building is never canceled automatically.
* `jobs` manages an app's scheduled jobs: fresh-container or exec cron jobs with run history, logs, retries and catch-up (see [Scheduled jobs](/scheduled-jobs)). `apps scheduled-tasks` is the older exec-only form of the same resource.
* `functions` runs an image as a function: it sleeps when idle, and a request to a sleeping one waits through the cold start (see [Functions](/functions)).
* `apps sleep` stops an app after it has had no requests for a set time and wakes it on the next request (see [Sleep when idle](/sleep-when-idle)).
* `apps canary` runs a new image beside the app and sends it a share of traffic until you promote or abort it (see [Canary deploys](/canary-deploys)).
* `apps auto-update` redeploys an app automatically when its image tag moves to a new digest (see [Image auto-update](/image-auto-update)).
* `apps auto-rollback` opts an app into or out of automatic rollback when a crashloop alert fires. `apps auto-rollback-slo-burn` sets how the app reacts when an SLO burn rate alert fires.
```
levelrail-cli deployments list [--status a,b] [--app NAME] [--branch B] [--trigger T] [--environment E] [--since 24h] [--until T] [--q TEXT] [--live] [--pr N] [--limit N] [--cursor C] [flags]
levelrail-cli deployments summary [--window 24h] [flags]
levelrail-cli deployments watch [flags]
```
Deploys across every app you can read: a filterable newest first list (page size default 50, max 200, with `--cursor` paging), counts by status with failure rate and duration percentiles, and a live event stream (`--json` prints one object per event).
```
levelrail-cli deploy-approvals list [--status pending|all|approved|rejected|expired] [--service NAME] [flags]
levelrail-cli deploy-approvals get [flags]
levelrail-cli deploy-approvals approve [flags]
levelrail-cli deploy-approvals reject [--reason TEXT] [flags]
```
A deploy or promote into an environment tagged protected becomes a pending approval that a different, sufficiently privileged user must approve first; the same user or token that requested it cannot decide it. `list` defaults to pending. `approve` runs the gated deploy or promote now; `reject` leaves the app's desired state untouched.
```
levelrail-cli apps deploy-notify-targets list [flags]
levelrail-cli apps deploy-notify-targets create --channel-id ID [flags]
levelrail-cli apps deploy-notify-targets delete [flags]
```
Attach an already connected notification channel (see [Channels](#channels)) so a deploy success or failure sends a message there.
### Domains, ports and traffic
```
levelrail-cli apps domains list [flags]
levelrail-cli apps domains add ... [--dns auto|off|preview] [--replace] [--wait] [--timeout 2m] [flags]
levelrail-cli apps domains remove ... [--remove-dns] [flags]
levelrail-cli apps streams list [flags]
levelrail-cli apps streams create --container-port N --host-port N [--protocol tcp] [flags]
levelrail-cli apps streams delete [flags]
levelrail-cli apps egress get [flags]
levelrail-cli apps egress set --allow host:port [--allow host:port ...] [flags]
levelrail-cli apps egress clear [flags]
levelrail-cli apps exec-access enable|disable|status [flags]
```
* `apps domains add` creates the DNS record when a provider is connected (`--dns off` skips it, `--dns preview` shows it); `--wait` prints the go-live steps and exits non-zero when the domain is not live within `--timeout`. See [Add a domain and go live](/domains-and-ingress#add-a-domain-and-go-live).
* `apps domains` shows or changes an app's domains; a domain already used by another app is refused and nothing is changed. Per-domain TLS, redirects, WAF and the like are under [Domains](#domains).
* `apps streams` forwards a host port to one container port as raw TCP (only `tcp` is supported). A stream change takes effect on the app's next container recreation, such as an image change or `apps restart`.
* `apps egress` restricts an app's outbound traffic to the declared `host:port` pairs; with nothing configured, egress is unrestricted. DNS and loopback traffic stay open regardless.
* `apps exec-access` is on by default. When disabled, `apps exec` and the interactive terminal are refused whatever abilities the token has.
### Run, logs and metrics
```
levelrail-cli apps exec -- [args...] [flags]
levelrail-cli apps logs [flags]
levelrail-cli apps metrics --metric NAME [flags]
levelrail-cli apps requests [flags]
levelrail-cli apps resource-usage [flags]
levelrail-cli apps resource-recommendation [flags]
levelrail-cli apps cost [flags]
levelrail-cli apps health-score [flags]
levelrail-cli apps log-drain get ]