> ## Documentation Index
> Fetch the complete documentation index at: https://docs.levelrail.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts

> Alert rules, the inbox, history, silences, maintenance windows and the 18 notification channel kinds, with the dashboard, CLI and API steps for each.

An alert is a rule the control plane evaluates, plus a notification channel it sends to when the rule fires. Everything runs inside the control plane, so no outside service is needed to evaluate rules.

## What can trigger an alert?

A rule has a kind, and the kind decides what it watches. The kinds in `internal/alerting`:

* Apps: `threshold` (a metric, a comparator and a value, held for a duration), `crashloop`, `oom_killed`, `slo_burn`, `log_pattern`, `log_level_rate`, `deploy_failed`.
* Jobs: `scheduled_task_failure`, `scheduled_task_missed`, `scheduled_task_running_long`, `job_failing`. See [Scheduled jobs](https://docs.levelrail.com/scheduled-jobs.md).
* Certificates and domains: `cert_expiry`, `cert_expiring`, `cert_renewal_stalled`, `parked_domain_cert_expiry`, `domain_health`, `domain_not_resolving`, `domain_traffic_drop`.
* Nodes and resource pressure: `node_offline`, `node_disk_space`, `node_host_guardrail`, `node_resource_usage`, `node_cpu`, `node_memory`, `node_cert_expiring`, `volume_usage`, `patch_status`, `version_skew`. See [Host pressure](https://docs.levelrail.com/host-pressure.md).
* Databases: `database_down`, `database_connections_high`, `database_disk_high`, `database_backup_failing`, `database_backup_stale`, `pooler_saturated`.
* Backups and exports: `backup_failed`, `backup_missing`, `backup_copy_behind`, `control_plane_backup_stale`, `restore_drill_failed`, `log_archive_stale`, `export_behind`.
* Platform: `registry_refresh_failing`, `email_delivery_failing`, `cleanup_failing`, `budget_threshold`.
* Uptime: `monitor_down`, which reports a down [monitor](https://docs.levelrail.com/monitors.md). Monitors are experimental and off unless `APP_MONITORS_ENABLED` is true.

The dashboard rule dialog, the CLI and the API do not accept exactly the same list. The API accepts every kind above. The CLI `apps alerts create` accepts the kinds named in its `--kind` help (see [CLI reference](https://docs.levelrail.com/cli-reference.md)).

## How do I create a rule from the dashboard?

Open the app's Alerts section and choose **Create alert rule**. Pick a kind, fill the fields that kind needs, attach a notification channel and save. Platform-wide kinds (for example `node_offline` or `control_plane_backup_stale`) are not tied to one app, and their resource is only a display label.

Optional per-rule settings are a severity (`info`, `warning` or `critical`), labels that silences can match, a consecutive failure count, and a flap threshold and window. Left empty, the control plane defaults apply.

## How do I create a rule from the CLI or API?

Rules belong to an app and live under `apps alerts`:

```bash theme={"dark"}
levelrail-cli apps alerts create web --name "High CPU" --kind threshold --metric cpu_percent --comparator ">" --threshold 90 --for-duration 5m --channel-id CHANNEL_ID --severity warning
levelrail-cli apps alerts create web --name "Restart loop" --kind crashloop --restart-count-threshold 3 --restart-window 5m --channel-id CHANNEL_ID
levelrail-cli apps alerts list web
levelrail-cli apps alerts update web RULE_ID --kind threshold --metric cpu_percent --comparator ">" --threshold 95
levelrail-cli apps alerts delete web RULE_ID
levelrail-cli apps alerts template web --profile-type web --channel CHANNEL_ID
```

`threshold` needs `--metric`, `--comparator` and `--threshold`. `crashloop` needs `--restart-count-threshold` and `--restart-window`. Run `levelrail-cli apps alerts create -h` for the other flags.

API, with `write` ability to change and `read` to list:

| Method and path | Purpose |
| - | - |
| `GET`, `POST /api/v1/apps/{name}/alerts` | List or create rules |
| `PUT`, `DELETE /api/v1/apps/{name}/alerts/{id}` | Update or delete a rule |
| `POST /api/v1/apps/{name}/alerts/{id}/silence` | Silence one rule |
| `POST /api/v1/apps/{name}/alerts/apply-template` | Apply an alert template |
| `GET /api/v1/alert-templates` | List alert templates |

A rule body carries `name`, `kind`, `channel_id`, `enabled`, `severity`, `labels` and the kind fields such as `metric`, `comparator`, `threshold` and `for_duration`. See the [API reference](https://docs.levelrail.com/api-reference.md).

## Which alerts exist by default?

A set of platform-wide rules is seeded on first boot, and you can turn each on or off. They include certificate expiry at 14, 7, 3 and 1 days (`APP_DEFAULT_ALERT_CERT_DAYS` changes the days), node offline, node disk space, control plane backup stale, restore drill failed, backup copy behind, export destination failing, log archive failing, registry credential refresh failing, outbound email failing, data cleanup failing, a failed deploy, a failed database backup, a stale database backup and a database down. A `monitor_down` default is added when `APP_MONITORS_ENABLED` is true.

In the dashboard they are in **Alerts, Defaults**, with a switch per alert and a reset. From the CLI:

```bash theme={"dark"}
levelrail-cli alerts defaults list
levelrail-cli alerts defaults disable ID
levelrail-cli alerts defaults enable ID
levelrail-cli alerts defaults reset
```

API: `GET /api/v1/alert-defaults`, `PUT /api/v1/alert-defaults/{id}`, `POST /api/v1/alert-defaults/reset`.

## Where do I see what fired?

The Alerts page has five tabs: Inbox, History, Silences, Maintenance and Defaults.

* Inbox: alerts that reached you, with per-account read state, routing and an optional daily digest. See [Alert inbox](https://docs.levelrail.com/alert-inbox.md).
* History: every firing and resolution, and what happened to each notification. Outcomes are `sent`, `silenced`, `grouped`, `inhibited`, `failed`, `ratelimited`, `flapping` and `skipped`. History is kept for 30 days by default (runtime setting `retention.alert_history_days`, seeded by `APP_ALERT_HISTORY_RETENTION`).

```bash theme={"dark"}
levelrail-cli alerts history --app web --outcome failed --limit 20
levelrail-cli alerts history --changes
```

`--changes` adds what changed on the app before each firing. API: `GET /api/v1/alert-history` and `GET /api/v1/apps/{name}/alert-history`.

## How do I mute an alert?

A silence stops matching alerts from notifying while they keep evaluating and stay in history. Every filter you give must match.

```bash theme={"dark"}
levelrail-cli alerts silence web RULE_ID --for 4h --reason "planned migration"
levelrail-cli alerts silences create --for 2h --node node-1 --severity warning
levelrail-cli alerts silences list
levelrail-cli alerts silences delete ID
```

Filters are rule, app, node, kind, severity, metric and label. API: `GET`, `POST /api/v1/silences` and `DELETE /api/v1/silences/{id}`.

## How do I mute alerts during planned work?

Create a maintenance window: a recurring silence defined by a cron expression, a duration and a time zone, scoped to all apps and nodes, to apps or to nodes.

```bash theme={"dark"}
levelrail-cli alerts maintenance create --name "Weekly patching" --cron "0 3 * * 0" --duration 2h --tz Europe/Berlin --scope node --target node-1
levelrail-cli alerts maintenance list
levelrail-cli alerts maintenance update ID --disabled
levelrail-cli alerts maintenance delete ID
```

API: `/api/v1/alert-maintenance-windows`. For one-off planned downtime shown on the dashboard and the public status page, use [Scheduled maintenance](https://docs.levelrail.com/scheduled-maintenance.md) instead.

## How do I get notified in Slack?

Connect a Slack channel once, then attach it to rules. The same steps work for every kind.

1. Open **Settings, Notification channels** and choose **Connect channel**.
2. Pick the kind, name it and paste the destination (for Slack, an incoming webhook URL).
3. Send a test, then attach the channel to a rule.

```bash theme={"dark"}
levelrail-cli channels create --name team-slack --kind slack --notify-url https://hooks.slack.com/services/XXX
levelrail-cli channels test CHANNEL_ID
levelrail-cli channels deliveries CHANNEL_ID
levelrail-cli channels quiet-hours CHANNEL_ID --start 22:00 --end 07:00 --timezone Europe/Berlin
```

`test` sends a real message and `deliveries` lists the recorded send history. Quiet hours hold back non-critical firing notifications on one channel. A disabled channel also silences the rules attached to it. API: `/api/v1/notification-channels`, with `/test`, `/deliveries` and `/quiet-hours`.

## Which notification channels are supported?

The code defines 18 kinds. Maturity is from [Feature status](https://docs.levelrail.com/feature-status.md).

| Kind | Label in the dashboard | Maturity |
| - | - | - |
| `generic` | Generic webhook | stable |
| `email` | Email | stable |
| `slack` | Slack | beta |
| `discord` | Discord | beta |
| `telegram` | Telegram | beta |
| `pushover` | Pushover | beta |
| `pagerduty` | PagerDuty | beta |
| `teams` | Microsoft Teams | beta |
| `resend` | Resend | beta |
| `ntfy` | ntfy | beta |
| `gotify` | Gotify | beta |
| `mattermost` | Mattermost | beta |
| `lark` | Lark | beta |
| `rocketchat` | Rocket.Chat | beta |
| `opsgenie` | Opsgenie | beta |
| `webex` | Webex | beta |
| `googlechat` | Google Chat | beta |
| `webpush` | Browser push | beta |

Beta means tested against local fakes that check each vendor's documented payload shape, with no real vendor account accepting a send yet. `webpush` is tested with a recording sender only, and its destination is every registered browser subscription instead of a URL. `channels create` accepts every kind except `webpush`. For email providers see [Email notifications](https://docs.levelrail.com/email-notifications.md).

## How does Levelrail avoid alert noise?

Several controls sit between a firing rule and the channel:

* Consecutive failures: a rule fires only after its condition holds for this many evaluations in a row. Default 1 (`APP_ALERT_CONSECUTIVE_FAILURES`), or set per rule.
* Flap detection: a rule that fires 5 times in 30 minutes is marked flapping (`APP_ALERT_FLAP_THRESHOLD`, `APP_ALERT_FLAP_WINDOW`, or per rule).
* Grouping: firing alerts for one target can be buffered into one summary. Off by default (`APP_ALERT_GROUP_WINDOW` is 0). A repeat of the same rule replaces its entry in the group.
* Rate limit: at most 30 sends per 10 minutes per target (`APP_ALERT_CHANNEL_RATE_LIMIT`, `APP_ALERT_CHANNEL_RATE_WINDOW`). Over the limit, the send is recorded as `ratelimited`.
* Inhibition: alerts for apps on a node that is offline are recorded as `inhibited`, so the node alert is the one you see.
* Silences, maintenance windows and channel quiet hours, above.

Noise control state is held in memory, so a control plane restart forgets pending groups and streaks. Severity is `info`, `warning` or `critical`. The inbox hides `info` by default.

## How do alerts relate to the status page and heartbeats?

They are separate. Alerts notify you. The [status page](https://docs.levelrail.com/status-page.md) is what your users see, and only monitors with the status link setting open and resolve its incidents. The [off-host heartbeat](https://docs.levelrail.com/heartbeat.md) is a ping an outside service watches, so you hear about a dead control plane that cannot send alerts itself. [External monitoring](https://docs.levelrail.com/external-monitoring.md) compares the three.

Deploy results can also go to a channel with `levelrail-cli apps deploy-notify-targets create <app> --channel-id ID`, and signed JSON events go out through [Event webhooks](https://docs.levelrail.com/event-webhooks.md).

## Related

* [Alert inbox](https://docs.levelrail.com/alert-inbox.md)
* [Observability](https://docs.levelrail.com/observability.md)
* [Notification channel email setup](https://docs.levelrail.com/email-notifications.md)
* [Monitors](https://docs.levelrail.com/monitors.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.