Runbooks

One page per alert family in deploy/monitoring/prometheus-rules.yaml. Every alert's runbook_url points at one of these, and every page lists its alerts on an Alerts: line. Each page has the same four sections: Symptom (what the operator or the provider sees), Metric (what to look at to tell causes apart), Action (what to do, cause by cause) and Verification (how to know it worked). mise run check:monitoring fails when an alert has no page, a page lacks a section, or the two disagree.

The dashboard is deploy/monitoring/grafana/ankusa.json; its sections are named for the runbooks they lead to. The thresholds are the SLO table's (architecture), and how to load all three is in deployment.

Runbook	Alerts	What it covers
ack-slo	AnkusaAckErrorRatioHigh, AnkusaAckLatencySLOBreach, AnkusaAckLatencyDegraded, AnkusaAckLatencyCritical, AnkusaIngestNearCapacity, AnkusaDrainBudgetAtRisk	503s, slow acks, capacity, the SIGTERM drain's commit budget
backlog-age	AnkusaBacklogStale, AnkusaDeliveryLatencySLOBreach	deliveries reaching their sinks late
breaker-open	AnkusaBreakerOpen	a sink's circuit breaker open
dlq-growth	AnkusaDLQGrowing, AnkusaDLQFull	dead letters piling up; the DLQ ceiling refusing hooks
disk-full	AnkusaDiskLow	the data volume running out
backup-stale	AnkusaBackupStale	the checkpoint upload failing or not running
store-write-failed	AnkusaStoreWriteFailures, AnkusaStoreUnavailable	the store failing writes, or closed
node-unready	AnkusaSubtreeDown	a failure domain out of restart budget; reading /ready
intake-down	AnkusaIntakeFailures, AnkusaIntakeNotReady, AnkusaIntakeReconcileStuck	the intake log of wal.type: dual
tenant-overload	AnkusaTenantOverload	one tenant refused with 429 tenant_overload
memory-growth	AnkusaMemoryGrowth	the VM's memory climbing
metrics-cardinality	AnkusaMetricsSeriesOverBudget	more than 10 000 series on a node