Skip to main content
ACME issuer configuration, settings toggles, and form validation are covered by automated tests: config-shape unit tests in internal/ingress/acme_test.go and end-to-end tests against a local CA in internal/ingress/acme_live_test.go. Those tests cannot prove issuance against a real public CA with real DNS and real reachability on ports 80 and 443. This runbook is that verification. A first run is recorded at the bottom of this page, so you do not need to repeat it before trusting the feature. Repeating it on your own host is the fastest way to confirm your firewall is right. For a version that needs no DNS, use the Enable HTTPS card described in Domains and ingress.

​
Prerequisites

  • A domain you control, with access to create an A or AAAA record.
  • A control plane reachable from the public internet (installed with install.sh, as levelrail.service, or in Docker).
  • DNS already pointing at the server’s public IP and propagated.
  • Ports 80 and 443 open from the internet. HTTP-01 is the check where the CA requests a file over port 80 to prove you own the domain, so a cloud firewall, a router without port forwarding or ufw in default-deny will break issuance while looking fine from the server itself. Wildcards need DNS-01 instead.
  • A root API token or admin dashboard access, since saving ingress settings (PUT /api/v1/settings/ingress) needs the root ability.

​
Steps

1

Point DNS at the server

Wait for propagation. From a machine outside your network, dig +short your-domain.example.com should return the server’s public IP. Do not continue until it does: issuing before DNS propagates is the most common failure, and each failed attempt counts against the CA’s rate limit.You can also ask the control plane directly:
2

Set the primary domain

Open Domains (Infrastructure section of the sidebar, or /domains) and set Primary domain to your hostname, for example dashboard.example.com. After saving, a Certificate status badge and a DNS check panel appear. Use the panel to confirm the platform resolves your record before enabling ACME.
3

Turn on ACME, staging first

Switch on Issue real ACME certificates and fill in:
  • Contact email: required. The CA uses it to reach you about problems such as forced revocation. Use a monitored address.
  • Directory URL (optional): leave blank for Let’s Encrypt production.
For a first run, set the directory URL to Let’s Encrypt staging, https://acme-staging-v02.api.letsencrypt.org/directory. Staging certificates trigger browser warnings but exercise the same mechanism without touching production rate limits. When staging succeeds, clear the field and save again for a trusted certificate. Saving drops the previous issuer’s stored certificates so the new CA re-issues them.From the CLI:
4

Wait for issuance

Saving does not issue immediately. Ingress reconciles continuously, and any routed domain qualifies for a certificate once ACME is on and an email is set, so issuance normally starts within moments. The next section says what success looks like.

​
Confirm it worked

  • Dashboard: the Certificate status badge reads Healthy. The issuer names your CA (Let's Encrypt, or an intermediate such as R11), never Levelrail’s internal issuer. The same data is at GET /api/v1/certificates and levelrail-cli domains certificates.
  • Expiry: not_after is about 90 days out for Let’s Encrypt. A lifetime of a few hours means you are still on the internal self-signed issuer: the toggle did not save, or the page is stale.
  • Renewal: nothing to configure. Caddy renews well before expiry (about 30 days out for a 90-day certificate). To see it work without waiting, look in the logs for renewing certificate or certificate obtained successfully before the current not_after.
  • Logs: journalctl -u levelrail -n 200 --no-pager, or your container’s logs. Look for "logger":"tls.obtain" and "logger":"http.acme_client".
An error between “trying to solve challenge” and “authorization finalized” is almost always one of the DNS or port 80 failures below, not an issuer bug.

​
Common failures

The log shows a challenge failure such as connection ... timed out or no such host right after “trying to solve challenge”.Check dig +short your-domain.example.com from outside your network, wait out the record’s TTL, then retry.
dig resolves correctly, but the log shows the challenge request timing out or being refused. Typical causes are cloud security groups, ufw, routers without port forwarding, and another proxy holding port 80.Open ports 80 and 443 inbound. levelrail-cli doctor and the dashboard’s status page include reachability checks. See Firewall: ports 80 and 443.
Let’s Encrypt production limits failed validations per account and hostname, and certificates per registered domain per week. Check their current limits, because they change. The log shows an explicit rate limited or too many error. Use staging for repeated attempts. Production enable attempts from the one-click flow are also capped by APP_ACME_MAX_ATTEMPTS_PER_HOUR.
The dashboard form rejects this before it reaches the API, and the backend rejects it the same way if you call the API directly.
Enabling ACME, or changing the directory URL (for example staging to production), drops the previous issuer’s stored certificates when you save so the new CA re-issues them. If a domain still shows the wrong issuer, confirm an app actually routes that domain: ACME applies only to domains in active use by a route.

​
Recorded run (2026-10-05)

Host: a 4 GB DigitalOcean droplet running the control plane from the public install.sh, ports 80 and 443 open to the internet, no DNS records created. Hostname: <dashed-ip>.sslip.io for the dashboard and for every app without a domain. What was done and seen:
  • Staging first: POST /api/v1/settings/ingress/https with staging: true. HTTP-01 validated, certificate issued by (STAGING) ..., shown by GET /api/v1/certificates.
  • Production: Let’s Encrypt (acme-v02) issued <dashed-ip>.sslip.io and the generated hostnames of three apps in one batch. Checked from a second server, with its normal system trust store and no -k: curl verify result 0, issuer Let's Encrypt (YE2), not_after 90 days out, SAN equal to the hostname.
  • http://<host>/ answered 308 to https://<host>/.
  • HSTS was enabled and disabled from the settings toggle.
What the run found and fixed:
  1. Certificates were stored on disk, not in the database as this page and ADR 005 describe, so GET /api/v1/certificates, the certificate expiry alert rules and the issuance audit trail all saw nothing. Storage is now the database, and existing on-disk certificates are imported on first start.
  2. Port 80 was closed outside ACME challenges, so any plain http:// request was refused instead of redirected.
  3. Changing the CA never took effect on a live reload, because certmagic reuses a valid certificate from any issuer. Saving now drops the other issuers’ certificates.
  4. Strict-Transport-Security was not sent on the dashboard’s HTML page, only on API responses.
  5. A failed issuance was invisible: only a log line. The CA’s error is now captured from Caddy’s certificate events and returned with a hint code.
Not covered by this run: the live capture of a CA error (item 5 is covered by unit tests only), renewal (certificates are 90 days old at best, so the renewal path ran only in tests and the got renewal info log lines), wildcard certificates through DNS-01, and a deliberately failing issuance against the production CA.

​
See also

Last modified on October 12, 2026