Concepts

Observability is the ability to measure a system’s current state based on the telemetry data it generates, such as traces, metrics, logs and events.

An observable system may be queried to describe its composition and behaviour. As systems are often highly distributed in nature, observability addresses the need to understand the behaviour with context across services, technologies and environments.

Monitoring is a complementary activity to observability, where typically a dashboard is configured to report predetermined metrics and alerts are set to trigger on specific conditions in the system. Monitoring is oriented to the detection of known types of issues, as the monitored metrics or conditions are determined in advance.

Observability enables detection and exploration of issues as they occur and assists with root cause analysis. By querying an observable system, unexpected types of issue can be detected, characterised and monitored as they arise, addressing the problem of "unknown unknowns". Questions on the actual behaviour of an observable system may be more readily answered:

Through enhanced visibility of system behaviour, observability improves understanding of the impact of changes made by developers and operators.

OpenTelemetry is a CNCF incubating project formed after the merger of OpenTracing and OpenCensus.

OpenTelemetry is a set of APIs, SDKs, tooling, integrations and semantic conventions for creating and managing telemetry data. OpenTelemetry provides a vendor-agnostic implementation per language (Java, Go, JavaScript, C++ and others) that exports telemetry data to backends of your choice. The Java implementation supports automatic instrumentation of many libraries, frameworks and application servers and manual instrumentation of library or application source code. OpenTelemetry supports various backends, including OSS projects (Jaeger, Zipkin, Prometheus, Grafana Tempo, SigNoz) and commercial products (Honeycomb, Lightstep, Aspecto, Dynatrace, New Relic, Datadog and more).

The OpenTelemetry project also provides the OpenTelemetry Collector, a vendor-agnostic deployable process which receives, processes and exports telemetry data.

Work on the OpenTelemetry specification and implementations is in progress by a large community of contributors. Support for traces is the most mature, followed by metrics and logs. See a summary of OpenTelemetry status.

There is a growing consensus in the commercial monitoring and APM tools market to adopt OpenTelemetry as a standard for telemetry in cloud-native software. Today many commercial products accept OpenTelemetry trace data. Product support for metrics and logs is growing. Several vendors have created distributions of the OpenTelemetry SDK and OpenTelemetry Collector, which feature customisations tailored for use with their products.

clj-otel extends the reach of OpenTelemetry to Clojure libraries and applications by providing:

clj-otel is an umbrella project for several Clojure modules clj-otel-*. They depend on the OpenTelemetry implementation for Java opentelemetry-java and the OpenTelemetry instrumentation agent provided by opentelemetry-java-instrumentation.

A trace represents the flow of a single transaction throughout the system. A trace comprises a tree of spans, where a span represents a unit of work in a service. Parent-child relationships between spans describe dependencies between them. The root span of a trace typically describes the entire transaction. The other spans represent units of work performed as part of the transaction. Traces provide context for system activity performed in spans.

Span data may include a span kind, name, attributes, start/end timestamps, links to other spans, a list of events and a status.

See specifications for span and span kind.

A metric is a numerical measurement over a period of time. Metrics may be used to indicate quantitative aspects of system health, such as resource (memory, disk, compute, network) usage, error rate, message queue length, and request response time. They may also be used to record statistics on application usage, performance and internal operation.

An instrument is used to record measurements. Instruments may support synchronous (invoke) or asynchronous (callback) recording of measurements, depending on the type of instrument. Measurements have a value of type long or double, along with context and attributes.

The available types of instrument are:

A service log is made of lines of text (possibly structured e.g. in JSON format) written when certain points in the service code are executed. Logs are well suited to ad-hoc debugging and capture of low-level details.

Events are captured as either a specific type of log or as a span event. Events are records that describe actions taken by the system or significant environmental changes, such as a service deployment or change in configuration.

Attributes may be attached to some telemetry data such as spans and resources. Attributes are a map where each entry has a string key. Each entry value is a boolean, long, double, string or an array of one of those types. Entries with a nil value are dropped.

OpenTelemetry recommends using namespaced attribute names to prevent clashes. See the specification for attributes and attribute naming.

A resource captures information about the entity for which telemetry data is recorded. For example, information on the host and JVM version may be part of a resource. Resources form part of the telemetry data.

The OpenTelemetry SDK contains resource implementations which capture host and process information.

Baggage is a mechanism for propagating telemetry metadata and is represented as a simple map. It is a means to add contextual information at a point in a transaction, read by a downstream service later in the same transaction and then used as an element of telemetry data, e.g. an attribute. For example, a user identifier is put in the baggage to indicate the principal of a request and subsequent spans in the trace include a principal attribute.

The current context is a context referenced by a specific thread local variable, defined by opentelemetry-java. It is the default context value for many functions in clj-otel and opentelemetry-java. The current context is appropriate to use when manually instrumenting synchronous code.

The bound context is a context referenced by a binding of a specific Clojure dynamic var, declared by clj-otel. Its use is optional for libraries and applications using clj-otel. It is provided as a convenience when manually instrumenting asynchronous code. If used, it overrides the default (current) context used for clj-otel functions.

Context propagation is the mechanism used to transmit context values across API boundaries and threads. Context propagation enables traces to become distributed traces, joining clients to servers and producers to consumers. In practice HTTP request header values are injected and extracted using a text map propagator.

OpenTelemetry provides text map propagators for the following protocols:

The W3C Trace Context and W3C baggage header propagation protocols are the most commonly used.

Autoconfiguration of the OpenTelemetry SDK refers to configuration using system properties or environment variables. Configuration of the OpenTelemetry instrumentation agent uses the same mechanism.

See documentation for SDK autoconfiguration and instrumentation agent configuration.

The SDK can be programmatically configured, as an alternative to autoconfiguration. This is a fallback option if autoconfiguration lacks the desired options.

This project clj-otel provides a module clj-otel-sdk for configuring the SDK in Clojure, as well as other support modules clj-otel-exporter-*,clj-otel-extension-* and clj-otel-sdk-extension-* for programmatic access to various optional components.