Concepts

Observability is the ability to measure a system’s current state based on the telemetry data it generates, such as traces, metrics, logs and events.

An observable system may be queried to describe its composition and behaviour. As systems are often highly distributed in nature, observability addresses the need to understand the behaviour with context across services, technologies and environments.

Monitoring is a complementary activity to observability, where typically a dashboard is configured to report predetermined metrics and alerts are set to trigger on specific conditions in the system. Monitoring is oriented to detection of known types of issues, as the monitored metrics or conditions are determined in advance.

Observability enables detection and exploration of issues as they occur and assists with root cause analysis. By querying an observable system, unexpected types of issue can be detected, classified and monitored as they arise, addressing the problem of "unknown unknowns". Questions on the actual behaviour of an observable system may be more readily answered:

Through enhanced visibility of system behaviour, observability improves understanding of the impact of changes made by developers and operators.

OpenTelemetry is a CNCF incubating project, formed after the merger of OpenTracing and OpenCensus.

OpenTelemetry is a set of APIs, SDKs, tooling, integrations and semantic conventions that are designed for the creation and management of telemetry data. OpenTelemetry provides a vendor-agnostic implementation per language (Java, Go, JavaScript, C++ and others) that can be configured to export telemetry data to backends of your choice. The Java implementation supports both automatic instrumentation of a large number of libraries, frameworks and application servers and manual instrumentation of library or application source code. OpenTelemetry supports various backends, including OSS projects (Jaeger, Zipkin, Prometheus, Grafana Tempo, SigNoz) and commercial products (Honeycomb, Lightstep, Aspecto, Dynatrace, New Relic, Datadog and more).

The OpenTelemetry project also provides the OpenTelemetry Collector, a vendor-agnostic deployable process which receives, processes and exports telemetry data.

Work on the OpenTelemetry specification and implementations is in progress by a large community of contributors. Support for traces is the most mature, followed by metrics and then logs. See a summary of OpenTelemetry status.

There is a growing consensus in the commercial monitoring and APM tools market to adopt OpenTelemetry as a standard for telemetry in cloud native software. Many commercial products today offer ingest of OpenTelemetry trace data and support for metrics and logs is growing. Several vendors have created their own distributions of the OpenTelemetry SDK and OpenTelemetry Collector, which are customisations tailored for use with their products.

clj-otel extends the reach of OpenTelemetry to Clojure libraries and applications by providing:

clj-otel is an umbrella project for several Clojure modules clj-otel-*. They depend on the OpenTelemetry implementation for Java opentelemetry-java and the OpenTelemetry instrumentation agent provided by opentelemetry-java-instrumentation.

A trace represents the flow of a single transaction throughout the system. A trace comprises a tree of spans, where a span represents a unit of work in a service and the parent-child relationship between the spans represent dependencies between them. The root span of a trace typically describes the entire transaction and the other spans in the trace describe units of work performed as part of the transaction. Traces provide context for system activity performed in spans.

Span data may include a span kind, name, attributes, start/end timestamps, links to other spans, a list of events and a status.

See specifications for span and span kind.

A metric is a numerical measurement over a period of time. Metrics are used to indicate quantitative aspects of system health, such as resource (memory, disk, compute, network) usage, error rate, message queue length, and request response time.

A service log is made of lines of text (possibly structured e.g. in JSON format) written when certain points in the service code are executed. Logs are well suited to ad-hoc debugging and capture of low-level details.

Events are captured as either a specific type of log or as a span event. Events are records that describe actions taken by the system over time, or environmental changes that occurred which are significant to the system, such as a service deployment or change in configuration.

Attributes may be attached to some telemetry data such as spans and resources. Attributes are a map where each entry has a string key and a value which is a boolean, long, double, string or an array of one of those types. Attributes with nil values are dropped.

See the specification for attributes and attribute naming.

A resource captures information about the entity for which telemetry data is recorded. For example, information on the host and JVM version may be part of a resource. Resources are included as part of other telemetry data such as traces and metrics.

The OpenTelemetry SDK contains resource implementations which capture a variety of host and process information.

Baggage is mechanism for propagating telemetry metadata and is represented as a simple map. It is intended as a means to add contextual information at a point in a transaction, to be read by a downstream service later in the same transaction and then used as an element of telemetry data e.g. an attribute. For example, a user identifier may put in the baggage to indicate the principal of a request and subsequent spans in the trace include a principal attribute.

The current context is a thread local io.opentelemetry.context.Context object. It is used as a default for many functions in this project clj-otel and methods of the underlying Java library opentelemetry-java. The current context is safe to use when manually instrumenting synchronous code.

Context propagation is the mechanism used to transmit context values across API boundaries and threads. Context propagation enables traces to become distributed traces, joining clients to servers and producers to consumers. In practice, this is achieved by injecting and extracting header values in HTTP requests using a text map propagator.

OpenTelemetry provides text map propagators for the following protocols:

The W3C Trace Context and W3C baggage header propagation protocols are the most commonly used protocols for propagation of trace context and baggage.

Autoconfiguration of the OpenTelemetry SDK refers to configuration using system properties or environment variables. Configuration of the OpenTelemetry instrumentation agent uses the same mechanism.

See documentation for SDK autoconfiguration and instrumentation agent configuration.

The SDK can be programmatically configured, as an alternative to autoconfiguration. This is a fallback option if autoconfiguration lacks the desired options.

This project clj-otel provides a module clj-otel-sdk for configuring the SDK in Clojure, as well as other support modules clj-otel-exporter-*,clj-otel-extension-* and clj-otel-sdk-extension-* for programmatic access to various optional components.