Liking cljdoc? Tell your friends :D

observability

Logging, metrics, tracing and error reporting behind ports, so a service depends on a protocol and the provider is chosen in config.

What works, per provider

Each signal has its own Integrant key; :provider picks the adapter. An unknown provider logs a warning and falls back to :no-op.

Signal:providerWhat it does

Logging
:wagoe/logging

:no-op

Drops everything.

:stdout

Writes text or JSON lines to standard output (:format, :colors, :level).

:slf4j

Logs through clojure.tools.logging to Logback; :level is applied to Logback too.

Metrics
:wagoe/metrics

:no-op

Drops everything.

:prometheus

In-memory registry; GET /metrics serves the Prometheus text format.

:otlp

Pushes to an OpenTelemetry collector over OTLP/HTTP every :interval-ms.

:datadog-statsd

Sends DogStatsD lines over UDP to a Datadog agent (:host, :port).

Tracing
:wagoe/tracing

:no-op

Records nothing.

:logging

Logs span start, end and duration. Not exported.

:otlp

Exports spans to an OpenTelemetry collector over OTLP/HTTP.

Error reporting
:wagoe/error-reporting

:no-op

Drops everything.

:sentry

Sends exceptions and messages to Sentry, redacted, filtered and sampled.

What does not exist:

  • No OpenTelemetry log export. Logs go to stdout or Logback; ship them with your log collector. Logs carry no trace or span ids unless you put them in the context.

  • No Datadog logging, tracing or error provider in config. Datadog takes traces and metrics through an OpenTelemetry collector (:otlp). wagoe.observability.logging.shell.adapters.datadog/create-datadog-logging-component exists, but :wagoe/logging cannot select it.

  • No Sentry tracing or metrics. Sentry is error reporting only.

Provider limits:

  • :otlp metrics are push-only: export-metrics throws, and get-metric-values, reset-metrics! and set-export-interval! do nothing. flush! forces an export.

  • :prometheus keeps summaries as sum and count; quantiles are not computed.

Metrics

Register a metric, then emit to its handle. An emission to a handle that was never registered is dropped by every provider.

(require '[wagoe.observability.metrics.ports :as metrics])

(def orders (metrics/register-counter! m :orders.placed "Orders placed" {}))
(metrics/inc-counter! m orders 1 {:channel "web"})

metrics/increment, increment-by, gauge, observe and timed register the metric on first use, so they need no registration step. The wagoe.observability.metrics.core helpers are built on them:

(metrics/increment m "orders.placed" {:channel "web"})
(metrics/observe m "checkout.ms" 12.5 {})

Keep label values bounded: a label per user id or URL makes a series per value. No provider caps the series of a metric. A cap would drop data without saying so; the framework’s own labels (method, status, operation) are bounded.

Error reporting with Sentry

:wagoe/error-reporting
{:provider    :sentry
 :dsn         #env SENTRY_DSN
 :environment "prod"
 :release     #env APP_RELEASE
 :sample-rate 1.0                  ; fraction of events sent
 :max-breadcrumbs 100              ; newest kept, per thread
 :filtering   {:min-severity       :warning              ; messages below are dropped
               :exclude-exceptions ["java.io.IOException"] ; class name, full or simple
               :exclude-messages   ["healthcheck"]}}       ; substring

The platform reports HTTP 5xx errors only, once each. Context, tags and extra are PII-redacted, and database values quoted in exception messages are removed.

Context (breadcrumbs, user, tags, extra) is per thread: cleared at the start of each HTTP request, and emptied for each job, event handler and realtime handler, then restored.

The component also implements IErrorFilter and IErrorReportingConfig, and changes through either apply to the next event:

(require '[wagoe.observability.errors.ports :as errors])

(errors/add-filter-rule! reporter {:type :exception :pattern #"timeout" :action :ignore})
(errors/add-filter-rule! reporter {:type :message :pattern "audit" :action :report})
(errors/add-filter-rule! reporter {:type :exception :pattern "ArithmeticException"
                                   :action :sample :sample-rate 0.1})
(errors/set-sample-rate! reporter 0.5)
(errors/disable-reporting! reporter)

The first matching rule decides. A string pattern names an exception class or is a substring of a message; a regex is searched in "<class>: <message>" or in the message; a function gets the exception and context, or the message, level and context. A :context rule takes a function of the context. add-filter-rule! returns the rule id for remove-filter-rule!.

Other keys in the Sentry config schema, such as rate limits, retries and :before-send, are not read.

Testing

clojure -M:test:test/otel :observability

Can you improve this documentation? These fine people already did:
thijscreemers & Thijs Creemers
Edit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close