Guides — com.github.steffan-westcott/clj-otel-api 0.1.0-SNAPSHOT

Add telemetry to your library or an application

To add telemetry to a library or application, use automatic instrumentation and/or manual instrumentation as described in the following subsections.

Add automatic instrumentation when running an application

Follow the guide to run the application with the OpenTelemetry instrumentation agent to add automatic instrumentation.

Add manual instrumentation to your library or application code

Add project dependency :

deps.edn

{;; ...
 :deps {com.github.steffan-westcott/clj-otel-api {:mvn/version "0.1.0"}}}

Follow the guides in the remainder of this section to make changes to your library or application code to add manual instrumentation.

Create a synchronous span using the current context

Use steffan-westcott.clj-otel.api.trace.span/with-span! to wrap a body of forms in a span, where the current context is used to keep track of (potentially nested) spans:
```
(defn get-nums [args]
  (span/with-span! {:name "Getting numbers"}
    (fetch-nums args)))
```

Create a synchronous span using explicit context

Use steffan-westcott.clj-otel.api.trace.span/with-span-binding to wrap a body of forms in a span, where the context to use is passed in as the :parent option. The new context containing the new span is bound to context* in this example:
```
(defn get-nums [context args]
  (span/with-span-binding [context* {:parent context
                                     :name "Getting numbers"}]
    (fetch-nums context* args)))
```

Create an asynchronous span

Use steffan-westcott.clj-otel.api.trace.span/async-span to start a new span that ends when either success/failure callback respond/raise is evaluated:

(defn get-nums-async [context args respond raise]
  (span/async-span {:parent context
                    :name   "Getting numbers"
                    :kind   :client}
                   (fn [context* respond* raise*]
                     (fetch-nums-async context* args respond* raise*))
                   respond
                   raise))

Add attributes to a span

Use the :attributes option to add attributes when creating a span:

(defn user-info [user-id]
  (span/with-span! {:name "Getting user info"
                    :attributes {:user-id user-id}}
    (fetch-user-info user-id)))

Alternatively, use steffan-westcott.clj-otel.api.trace.span/add-span-data! including the :attributes option to add attributes to an existing span.

By default, the span in the current context is updated:

(defn user-info [user-id]
  (span/add-span-data! {:attributes {:user-id user-id}})
  (fetch-user-info user-id))

Use the :context option to specify the context containing the span to update:

(defn user-info [context user-id]
  (span/add-span-data! {:context context
                        :attributes {:user-id user-id}})
  (fetch-user-info context user-id))

Add an event to a span

Use steffan-westcott.clj-otel.api.trace.span/add-span-data! including the :event option to add an event to an existing span. The event may include attributes.

By default, the event is added to the span in the current context:

(defn complete-stage [job-state]
  (span/add-span-data! {:event {:name "Stage completed"
                                :attributes (select-keys job-state [:stage :status])}})
  (notify-watchers job-state))

Use the context: option to specify the context containing the span to add the event to:

(defn complete-stage [context job-state]
  (span/add-span-data! {:context context
                        :event {:name "Stage completed"
                                :attributes (select-keys job-state [:stage :status])}})
  (notify-watchers context job-state))

Add an exception event to a span

Events for exceptions thrown in spans which leave (escape) the span’s scope are automatically added to the span. This applies to synchronous and asynchronous spans. For async-span, exceptions are "thrown" using the raise callback function.

Use steffan-westcott.clj-otel.api.trace.span/add-exception! to add an event describing an exception to an existing span. The exception event may include attributes. Use this function to capture details about caught (non-escaping) exceptions.

By default, the exception event is added to the span in the current context:

(defn process-args [args]
  (try
    (parse-args args)
    (catch Throwable e
      (span/add-exception! e {:escaping? false
                              :attributes {:args args}})
      {:result :parse-error})))

Use the context: option to specify the context containing the span to add the exception event to:

(defn process-args [context args]
  (try
    (parse-args args)
    (catch Throwable e
      (span/add-exception! e {:context context
                              :escaping? false
                              :attributes {:args args}})
      {:result :parse-error})))

Work with HTTP client and server spans

The guides in this section describe semantic conventions support for HTTP client and server spans.

Use Ring middleware for server span support

Use Ring middleware steffan-westcott.clj-otel.api.trace.http/wrap-server-span to add HTTP server span support to a Ring handler.

The enabled support features vary with the selected middleware options. The middleware can be configured to work in applications that run with or without the OpenTelemetry instrumentation agent. It also supports both synchronous (1-arity) and asynchronous (3-arity) handlers.

An example using Jetty with server spans created by the agent:
```
(defn handler [request]
  ;; ...
  )

(def service
  (-> handler
      (trace-http/wrap-server-span {:create-span? false
                                    :server-name  "my-server"})))

(defonce server
  (jetty/run-jetty #'service {:port 8080 :join? false}))
```

Use Pedestal interceptors for server span support

Use interceptors steffan-westcott.clj-otel.api.trace.http/server-span-interceptors to add HTTP server span support to a Pedestal HTTP service.

The enabled support features vary with the selected interceptor options. The interceptors can be configured to work in applications that run with or without the OpenTelemetry instrumentation agent.

An example using Jetty with server spans created by the agent:
```
(def routes
  (route/expand-routes
    [[["/" (trace-http/server-span-interceptors {:create-span? false
                                                 :server-name  "my-server"})
       ;; ...
       ]]]))

(def service-map
  {::http/routes routes
   ::http/type   :jetty
   ::http/port   8080
   ::http/join?  false})

(defonce server
  (http/start (http/create-server service-map)))
```

Add route data to a server span

Use steffan-westcott.clj-otel.api.trace.http/add-route-data! to add the matched route to a server span.

By default, the route data is added to the span in the current context:
```
(trace-http/add-route-data! "/rooms/:room-id")
```
Use the context: option to specify the context containing the span to add the route data to:
```
(trace-http/add-route-data! "/rooms/:room-id" {:context context})
```

Manually add HTTP response data to a client span

When an application is run with the OpenTelemetry instrumentation agent, HTTP response data is automatically added to HTTP client spans for supported clients.

Use steffan-westcott.clj-otel.api.trace.http/add-client-span-response-data! to add HTTP response data to a client span. Use this function when working with an HTTP client not supported by the OpenTelemetry instrumentation agent.

By default, the HTTP response data is added to the span in the current context:
```
(trace-http/add-client-span-response-data! response)
```
Use the context: option to specify the context containing the span to add the HTTP response data to:
```
(trace-http/add-client-span-response-data! response {:context context})
```

Manually propagate context in an HTTP client request

When an application is run with the OpenTelemetry instrumentation agent, context is automatically propagated in HTTP client requests for supported clients.

Use steffan-westcott.clj-otel.context/->headers to get headers to merge (inject) with the other headers in the HTTP request to be issued, for the purpose of context propagation. Use this function when working with an HTTP client not supported by the OpenTelemetry instrumentation agent.

By default, the current context is propagated:
```
(let [context-headers (context/->headers)
      request' (update request :headers merge context-headers)]
  ;; ...
  )
```
Use the :context option to specify the context to be propagated:
```
(let [context-headers (context/->headers {:context context})
      request' (update request :headers merge context-headers)]
  ;; ...
  )
```

Configure and run an application with telemetry

The options below determine what telemetry data is exported from an application as it runs. Select one of these options and follow the linked guide:

Run with the OpenTelemetry auto-instrumentation agent

Select this option to export telemetry data as a result of automatic and manual instrumentation.
Run with autoconfigured SDK

Select this option to export telemetry data from manual instrumentation only; The SDK will be configured using properties or environment variables.
Run with programmatically configured SDK

Select this option to export telemetry data from manual instrumentation only; The SDK will be configured programmatically in the application.
Run without agent or SDK

Select this option to mute all telemetry data.

Traces telemetry data is muted in the last option, or by setting the property otel.traces.exporter to none (the default is otlp) when using either of the first two options (autoconfiguration). Note that for metrics telemetry data, otel.metrics.exporter has default none.

Run with the OpenTelemetry auto-instrumentation agent

Download the latest version of the OpenTelemetry instrumentation agent JAR, the file opentelemetry-javaagent.jar from the releases page. The agent JAR includes the SDK and all its dependencies.
Configure the agent and SDK, using properties and environment variables. See the agent and SDK configuration documentation.
When running the application, enable the agent with the -javaagent JVM flag.

For an example application my-app, with deps.edn to export traces using OTLP over gRPC, use an alias like the following:

deps.edn

{;; ...
 :aliases {
   :otel {:jvm-opts ["-javaagent:path/to/opentelemetry-javaagent.jar"
                     "-Dotel.resource.attributes=service.name=my-app"
                     "-Dotel.traces.exporter=otlp"
                     "-Dotel.exporter.otlp.traces.protocol=grpc"]}}}

Run with autoconfigured SDK

Add project dependencies:
- Required: io.opentelemetry/opentelemetry-sdk-extension-autoconfigure for the SDK itself and SDK autoconfiguration.
- Required: io.opentelemetry/opentelemetry-exporter-??? for any exporters referenced in the configuration. See Java exporter libraries supported by autoconfiguration.
- Optional: io.opentelemetry/opentelemetry-sdk-extension-resources for various resources to be automatically added to telemetry data.
- Optional: io.opentelemetry/opentelemetry-sdk-extension-aws for various resources describing the AWS execution environment to be automatically added to telemetry data.
- Optional: io.opentelemetry/opentelemetry-extension-aws for text map propagator implementing the AWS X-Ray Trace Header propagation protocol.
- Optional: io.opentelemetry/opentelemetry-extension-trace-propagators for text map propagators implementing OpenTracing Basic Tracers, Jaeger and B3 propagation protocols.
- Optional: io.grpc/grpc-netty-shaded, io.grpc/grpc-protobuf and io.grpc/grpc-stub to use Netty for gRPC transport rather than the default OkHttp (see example below). This is not needed if gRPC is not used by any exporters or the application.
Configure the SDK using properties and environment variables. See SDK autoconfigure configuration documentation.

For an example application my-app, with deps.edn to export traces using OTLP over gRPC with Netty transport, use an alias like the following:

deps.edn

{;; ...
 :aliases {
   :otel {:jvm-opts ["-Dotel.resource.attributes=service.name=my-app"
                     "-Dotel.traces.exporter=otlp"
                     "-Dotel.exporter.otlp.traces.protocol=grpc"]
          :extra-deps {io.opentelemetry/opentelemetry-sdk-extension-autoconfigure {:mvn/version "1.11.0-alpha"}
                       io.opentelemetry/opentelemetry-sdk-extension-resources     {:mvn/version "1.11.0"}
                       io.opentelemetry/opentelemetry-exporter-otlp-trace         {:mvn/version "1.11.0"}
                       io.grpc/grpc-netty-shaded                                  {:mvn/version "1.44.1"}
                       io.grpc/grpc-protobuf                                      {:mvn/version "1.44.1"}
                       io.grpc/grpc-stub                                          {:mvn/version "1.44.1"}}}}}

Run with programmatically configured SDK

Add project dependencies:
- Required: com.github.steffan-westcott/clj-otel-sdk for the SDK itself and a Clojure wrapper of SDK configuration
- Required: com.github.steffan-westcott/clj-otel-exporter-??? for Clojure wrapped versions of any exporters referenced in the configuration. See Clojure wrapped versions of exporters supported by autoconfiguration.
- Optional: com.github.steffan-westcott/clj-otel-sdk-extension-resources for Clojure wrapped versions of various resources to add to telemetry data.
- Optional: com.github.steffan-westcott/clj-otel-sdk-extension-aws for Clojure wrapped versions of resources describing the AWS execution environment.
- Optional: com.github.steffan-westcott/clj-otel-extension-aws for Clojure wrapped text map propagator implementing the AWS X-Ray Trace Header propagation protocol.
- Optional: com.github.steffan-westcott/clj-otel-extension-trace-propagators for Clojure wrapped text map propagators implementing OpenTracing Basic Tracers, Jaeger and B3 propagation protocols.
- Optional: io.grpc/grpc-netty-shaded, io.grpc/grpc-protobuf and io.grpc/grpc-stub to use Netty for gRPC transport rather than the default OkHttp (see example below). This is not needed if gRPC is not used by any exporters or the application.
At application start, use steffan-westcott.clj-otel.sdk.otel-sdk/init-otel-sdk! to configure and set an OpenTelemetry SDK instance as the global OpenTelemetry instance.
At application end, use steffan-westcott.clj-otel.sdk.otel-sdk/close-otel-sdk! to close down activities of the SDK instance.

For an example application my-app, with deps.edn to export traces using OTLP over gRPC with Netty transport, use deps like the following:

deps.edn

{;; ...
 :deps {com.github.steffan-westcott/clj-otel-sdk                      {:mvn/version "0.1.0"}
        com.github.steffan-westcott/clj-otel-sdk-extension-resources  {:mvn/version "0.1.0"}
        com.github.steffan-westcott/clj-otel-exporter-otlp-grpc-trace {:mvn/version "0.1.0"}
        io.grpc/grpc-netty-shaded                                     {:mvn/version "1.44.1"}
        io.grpc/grpc-protobuf                                         {:mvn/version "1.44.1"}
        io.grpc/grpc-stub                                             {:mvn/version "1.44.1"}}}

To configure the SDK at start and close down at end, the application could have functions like the following:

example/app.clj

(ns example.app
  (:require [steffan-westcott.clj-otel.exporter.otlp-grpc-trace :as otlp-grpc-trace]
            [steffan-westcott.clj-otel.resource.resources :as res]
            [steffan-westcott.clj-otel.sdk.otel-sdk :as sdk]))

(defn init-otel! []
  (sdk/init-otel-sdk!
    "my-app"
    {:resources [(res/host-resource)
                 (res/os-resource)
                 (res/process-resource)
                 (res/process-runtime-resource)]
     :tracer-provider
       {:span-processors
         [{:exporters [(otlp-grpc-trace/span-exporter)]}]}}))

(defn close-otel! []
  (sdk/close-otel-sdk!))

Run without agent or SDK

There are no steps to add dependencies or otherwise configure the application in order to run it without the agent or SDK.

An application run without the OpenTelemetry instrumentation agent or SDK will not export any telemetry data. Usage of the OpenTelemetry API (manual instrumentation) in the application will invoke no-op implementations.

Use the OpenTelemetry Collector

The OpenTelemetry Collector is used to manage telemetry data, as an alternative to applications exporting data directly to telemetry backends.

Deploy the OpenTelemetry Collector

See the getting started documentation.

Configure the OpenTelemetry Collector

See the configuration documentation.

This example Collector configuration has a traces pipeline where:

Trace data are received by the Collector as OTLP over gRPC
Memory usage in the Collector process is limited
Traces are batched in the Collector prior to export to the backend
Traces are exported by the Collector to the Jaeger backend on host jaeger

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 750
  batch:

exporters:
  jaeger:
    endpoint: jaeger:14250
    insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ jaeger ]