Data export

​

What you can export

​

Prometheus scrape

scrape_configs:
  - job_name: levelrail
    scheme: https
    metrics_path: /metrics
    authorization:
      credentials_file: /etc/prometheus/levelrail.token
    static_configs:
      - targets: ['levelrail.example.com']

​

Loki and Grafana quick start

1. Run Loki and Grafana wherever you like. For a trial on one host:
   docker network create obs
   docker run -d --name loki --network obs -p 3100:3100 grafana/loki:3.3.2
   docker run -d --name grafana --network obs -p 3000:3000 grafana/grafana:11.3.0
   
2. Loki on the same host or a private network is an internal address, which export refuses by default. On the control plane set APP_EXPORT_ALLOW_PRIVATE_NETWORKS=true and restart (see Environment variables). A public HTTPS Loki needs nothing.
3. Add the destination:
   levelrail-cli export destinations create --name loki --kind logs_loki \
     --endpoint http://loki.internal:3100 --app web --min-level info \
     --label env=prod
   levelrail-cli export destinations test <id>
   
   For Grafana Cloud or a multi-tenant Loki add --header X-Scope-OrgID=tenant or --header "Authorization=Basic <base64>". Header values are stored encrypted and are never returned by the API.
4. In Grafana add a Loki data source pointing at http://loki:3100, then query {job="levelrail", app="web"}.

​

Log destinations

• HTTP JSON lines: one JSON object per line: time, kind, app, stream, level, message and fields for structured lines. Content-Type: application/x-ndjson.
• Loki: an endpoint with no path gets /loki/api/v1/push appended.
• Syslog: tcp://host:port or tls://host:port, RFC 5424 with octet counting framing, facility local0. TLS verifies the server certificate against the system roots.

​

Audit log export

import hmac, hashlib, time

def verify(key: bytes, header: str, body: bytes, tolerance=300) -> bool:
    parts = dict(p.split("=", 1) for p in header.split(","))
    expected = hmac.new(key, parts["t"].encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"]) and abs(time.time() - int(parts["t"])) <= tolerance

​

Delivery semantics

• A cursor (timestamp and row id) is saved after every successful batch, so a restart resumes where it stopped.
• Delivery is at least once. A crash between a receiver accepting a batch and the cursor being saved sends that batch again, with the same id range.
• A failing destination holds the cursor and retries. Rows are never skipped, and the audit log itself is the buffer.

​

On demand download

levelrail-cli export audit --from 2026-10-01T00:00:00Z --to 2026-10-08T00:00:00Z --format csv > audit.csv

​

OTLP metrics

levelrail-cli export destinations create --name otel --kind metrics_otlp \
  --endpoint https://otel.example.com:4318 --header "Authorization=Bearer <token>"

​

Reliability and safety

• Every destination has an in-memory queue of APP_EXPORT_QUEUE_LINES entries. When it fills, the oldest entries are dropped and counted. There is no disk buffer, so a control plane restart loses queued log lines but not audit rows.
• Container log lines are handed to exporters through a non-blocking subscription. A slow destination cannot slow a container, the log store, a deploy or the API.
• A batch is retried with exponential backoff and jitter up to APP_EXPORT_MAX_ATTEMPTS times. Responses in the 4xx range other than 408 and 429 are not retried, because retrying cannot fix them.
• After APP_EXPORT_BREAKER_FAILURES consecutive failed batches a destination backs off for APP_EXPORT_BREAKER_COOLDOWN, then tries one batch.
• Endpoints are validated when saved and again at every connection, including redirects: private, loopback and link-local addresses are refused unless APP_EXPORT_ALLOW_PRIVATE_NETWORKS=true. Endpoints cannot embed credentials.
• Headers and signing keys are stored with envelope encryption and are never returned. The API shows header names and whether a key is set.
• Each destination shows its last success, last error, queue depth, dropped and failed counts, and breaker state, in the dashboard, with levelrail-cli export destinations status <id>, and as export_* metrics.

​

CLI

levelrail-cli export destinations list
levelrail-cli export destinations create --name N --kind K [--endpoint URL] [--app A] [--min-level L] [--header K=V] [--hmac-key KEY] [--storage-target ID] [--label K=V] [--interval S] [--backfill] [--disabled]
levelrail-cli export destinations test <id>
levelrail-cli export destinations status <id>
levelrail-cli export destinations enable|disable <id>
levelrail-cli export destinations delete <id>
levelrail-cli export audit [--from T] [--to T] [--format ndjson|csv]

​

Next steps