/metrics exposes too many seriesAlerts: AnkusaMetricsSeriesOverBudget
One node's /metrics carries more than 10 000 ankusa_ series. The objective is at most 10 000 per node, independent of the number of tenants: a scrape that grows with tenants or sources slows the node's scrape, costs the monitoring system memory, and in the worst case gets the target dropped by sample_limit.
count by (instance) ({__name__=~"ankusa_.+"}): the alert's number.topk(10, count by (__name__) ({__name__=~"ankusa_.+", instance="<node>"})).count(count by (tenant_id) (<metric>)), then source_id, tenant, sink.metrics.detail and metrics.max_series_per_metric in GET /v1/config: the policy in force.metrics.detail decides which identity labels a per-hook metric carries: tenant (default: tenant_id on counters and per-tenant gauges, none on histograms), source (tenant_id and source_id, on histograms too) or none. Set tenant or none unless you need more; every histogram label set is a dozen series.metrics.max_series_per_metric caps what one metric keeps: the first max - 1 label sets, then "other". Lower it to bound the worst case. A gauge past the cap drops tenants instead of folding them.tenant_id nor source_id nor tenant in it is a bug: no label may come from unauthenticated input. Open an issue with the metric name and the label, and drop the label at your scrape with metric_relabel_configs meanwhile./metrics from the metrics listener (metrics.enabled) so a scrape does not share the admin port.count by (instance) ({__name__=~"ankusa_.+"}) is under 10 000 on every node and does not move when a tenant is added.metrics_test.exs (bounded series under 10k sources) passes in CI: the series set is independent of tenant count.Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |