Alerting

Admin › Alerting turns fleet samples into alerts: you define where alerts go (notification channels), what triggers them (alert rules), and see what fired (recent alerts). Editing needs alerting:edit; deleting needs alerting:delete.

Notification channels

Add channel, give it a name, pick a type and switch it Enabled:

Send test delivers a test message so you know the channel works before you rely on it.

Channels are shared: promises can also notify them on incident changes.

Alert rules

Add rule, then:

1. Name and Severity (info, warning or critical).

2. Thresholds — set any of these; 0 turns one off:

   Threshold	Fires when
   Node mem %	A node's memory use exceeds this percentage
   Query GB	A single query holds more than this much memory
   Query min	A single query has run longer than this many minutes
   Parts ETA min	A table is projected to reach parts_to_throw_insert (inserts start failing with too many parts) within this many minutes

3. Deliver to channels — one or more channels.

4. Optionally AI auto-RCA on breach with a Model: when the rule fires, the Doctor investigates the affected connection and sends its report to the same channels, at most once per DOCTOR_AUTO_RCA_COOLDOWN_MINUTES (default 60) per server.

Only one fleet rule can be active at a time; disable the current one before enabling another.

How alerts fire

Rules are checked on every fleet sample (OBSERVE_FLEET_INTERVAL, default 30 s). An alert fires once when a condition goes from healthy to breaching, and doesn't repeat while it stays breached; the state survives pod restarts and failover, so a rollout doesn't re-send everything. A parts-pressure alert re-arms only once the projected time climbs well clear of the limit, to avoid flapping.

Recent alerts lists what fired. Clear entries older than 24 hours, 7 days or 30 days, or all of them.

Suggested starting rules

Tune them after a week of watching Fleet trends, so alerts and the dashboard agree.