> ## Documentation Index
> Fetch the complete documentation index at: https://docs.levelrail.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Traffic observability: rate, errors and duration

> Per app and per replica request rate, error rate and p50/p95/p99 latency measured at the ingress, with slow request samples and SLO error-budget burn.

Every app that serves traffic through a domain gets request rate, error rate and latency percentiles with no setup and no change to the app. The numbers are measured at the embedded ingress, kept on the node that served the requests, and queried on demand.

## What you see

Open an app and choose **Traffic** under **Observe**, or run `traffic <app>` in the CLI. For the window you pick (5 minutes, 1 hour or 24 hours):

| Figure | Meaning |
| - | - |
| Rate | Requests per second |
| Errors | 4xx and 5xx share of requests, and a count by status class |
| Duration | p50, p95 and p99 latency |
| Replicas | The same figures per upstream address, so one bad container stands out |
| Slowest routes | Route templates ranked by average duration |
| Slow and failed requests | A sample of 5xx, upstream failures and slow requests |
| Error budget | Burn rate and budget remaining when the app has an SLO alert |

## Bounded by design

Memory and storage stay flat as traffic grows.

* Latency uses a fixed 12 bucket histogram per app and per replica. Percentiles are estimated from the buckets, so they are accurate to the bucket width.
* Routes are normalized templates such as `/api/users/:id`. Ids, tokens, query strings and client addresses are never read or stored. The number of distinct routes and replicas kept per tick is capped, and the rest fall into an `other` bucket.
* The sample log keeps failed requests (5xx and upstream failures) and a fraction of slow ones, capped per tick and per app, and expires with a retention period.

## Settings

All settings are environment variables on the control plane or agent.

| Variable | Default | Effect |
| - | - | - |
| `APP_REQUEST_SLOW_MS` | `1000` | Duration at which a request counts as slow |
| `APP_REQUEST_SAMPLE_RATE` | `0.1` | Fraction of slow requests sampled. Failed requests are always eligible |
| `APP_REQUEST_SAMPLE_TICK_MAX` | `50` | Samples kept per app per sampling tick. `0` turns sampling off |
| `APP_REQUEST_SAMPLE_ROWS` | `5000` | Samples stored per app |
| `APP_REQUEST_SAMPLE_RETENTION` | `24h` | How long samples are kept |
| `APP_REQUEST_UPSTREAM_MAX` | `32` | Distinct replica labels kept per tick |
| `APP_REQUEST_ROUTE_MAX` | `50` | Distinct routes kept per tick |
| `APP_TRAFFIC_SLOW_ROUTE_MIN` | `5` | Requests a route needs before it can rank as slow |
| `APP_ALERT_REQUEST_WINDOW` | `5m` | Trailing window of the request alert metrics |

## Alerts

A threshold alert rule can watch three request metrics, so "error rate above X for Y" and "p99 above X for Y" need no new rule type:

| Metric | Value |
| - | - |
| `request_error_rate_5xx_pct` | 5xx share of requests, as a percentage |
| `request_p95_ms` | p95 latency in milliseconds |
| `request_p99_ms` | p99 latency in milliseconds |

A window with no requests neither fires nor resolves a rule. For error-budget alerting use an SLO burn rule, which this page's budget figure reads.

## API

`GET /api/v1/apps/{name}/traffic?window=5m|1h|24h` returns the totals, `status_classes`, `latency`, `replicas`, `slow_routes`, `samples` and, when an SLO burn rule exists, `slo`. It needs read access to the app. Time series for the charts come from `GET /api/v1/apps/{name}/requests`.

## CLI

```
traffic <app> [--window 5m|1h|24h] [--watch] [--interval 5s] [--json]
```

With `--watch --json` each refresh is printed as one JSON line.

## Limits

* Replica figures need the request to pass through a reverse proxy upstream. Static sites and requests answered by the ingress itself have no replica.
* Percentiles are histogram estimates. A p99 above 10 seconds reports the last bucket bound.
* The report is not available through the MCP server.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.