Liking cljdoc? Tell your friends :D

Guides

Add telemetry to your library or an application

To add telemetry to a library or application, use automatic instrumentation and/or manual instrumentation as described in the following subsections.

Add automatic instrumentation to an application at runtime

Add manual instrumentation to your library or application code at design time

  • Add project dependency :

    deps.edn
    {;; ...
 :deps {com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.1"}}}

  • Follow the guides below to add manual traces and metrics instrumentation.

Add manual traces instrumentation

Create synchronous span

Create a synchronous span using the current context
Create a synchronous span using the bound context
Create a synchronous span using explicit context

  • Use steffan-westcott.clj-otel.api.trace.span/with-span-binding to wrap a body of forms in a span, where the context to use is passed in as the :parent option. The new context containing the new span is bound to context* in this example:

    (defn get-nums [context args]
  (span/with-span-binding [context* {:parent context
                                     :name "Getting numbers"}]
    (fetch-nums context* args)))

Create asynchronous span

Create an asynchronous span using the bound context

  • Use steffan-westcott.clj-otel.api.trace.span/async-bound-span to start a new span, where the context to use is the bound context. The new span ends when either success/failure callback respond/raise is evaluated:

    (defn get-nums-async [args respond raise]
  (span/async-bound-span {:name "Getting numbers"
                          :kind :client}
                         (fn [respond* raise*]
                           (fetch-nums-async args respond* raise*))
                         respond
                         raise))
Create an asynchronous span using explicit context

  • Use steffan-westcott.clj-otel.api.trace.span/async-span to start a new span, where the context to use is passed in as the :parent option. The new spans ends when either success/failure callback respond/raise is evaluated:

    (defn get-nums-async [context args respond raise]
  (span/async-span {:parent context
                    :name   "Getting numbers"
                    :kind   :client}
                   (fn [context* respond* raise*]
                     (fetch-nums-async context* args respond* raise*))
                   respond
                   raise))

Add attributes to a span

  • Use the :attributes option to add attributes when creating a span:

    (defn user-info [user-id]
  (span/with-span! {:name "Getting user info"
                    :attributes {:system/user-id user-id}}
    (fetch-user-info user-id)))

  • Alternatively, use steffan-westcott.clj-otel.api.trace.span/add-span-data! including the :attributes option to add attributes to an existing span.

    By default, the span in the bound or current context is updated:

    (defn user-info [user-id]
  (span/add-span-data! {:attributes {:system/user-id user-id}})
  (fetch-user-info user-id))

    Use the :context option to specify the context containing the span to update:

    (defn user-info [context user-id]
  (span/add-span-data! {:context context
                        :attributes {:system/user-id user-id}})
  (fetch-user-info context user-id))

Add an event to a span

  • Use steffan-westcott.clj-otel.api.trace.span/add-span-data! including the :event option to add an event to an existing span. The event may include attributes.

    By default, the event is added to the span in the bound or current context:

    (defn complete-stage [job]
  (span/add-span-data! {:event {:name "Job stage completed"
                                :attributes {:service.workflow.job/stage (:stage job)}}})
  (notify-watchers job))

    Use the :context option to specify the context containing the span to add the event to:

    (defn complete-stage [context job]
  (span/add-span-data! {:context context
                        :event {:name "Job stage completed"
                                :attributes {:service.workflow.job/stage (:stage job)}}})
  (notify-watchers context job))

Add an exception event to a span

clj-otel automatically adds events to spans for thrown exceptions which leave (escape) the span’s scope. This behaviour applies to synchronous and asynchronous spans.

  • Use steffan-westcott.clj-otel.api.trace.span/add-exception! to add an event describing an exception to an existing span. Use this function to capture details about caught (non-escaping) exceptions.

    The exception event may include attributes, controlled by the :attributes option.

    By default, the exception event is added to the span in the bound or current context:

    (defn process-args [args]
  (try
    (parse-args args)
    (catch Throwable e
      (span/add-exception! e {:escaping? false
                              :attributes {:app/args args}})
      {:result :parse-error})))

    Use the :context option to specify the context containing the span to add the exception event to:

    (defn process-args [context args]
  (try
    (parse-args args)
    (catch Throwable e
      (span/add-exception! e {:context context
                              :escaping? false
                              :attributes {:app/args args}})
      {:result :parse-error})))

Add manual metrics instrumentation

  • See this OpenTelemetry guide to select the appropriate instrument type to use.

  • Follow the instructions below to create the instrument and take measurements synchronously or asynchronously, depending on the type of instrument.

Create and use an instrument to take measurements synchronously

Counter, up-down counter and histogram instruments support taking measurements synchronously.

  • Use steffan-westcott.clj-otel.api.metrics.instrument/instrument to create an instrument of the required type.

    (defonce foo-count
  (instrument/instrument {:name "app.foo-count"
                          :instrument-type :counter
                          :unit "{foo}"
                          :description "The number of foos counted"}))

(defonce segment-size
  (instrument/instrument {:name "app.segment-size"
                          :instrument-type :histogram
                          :unit "{byte}"
                          :description "The size of requested segment"}))

  • Use steffan-westcott.clj-otel.api.metrics.instrument/add! to add a measurement to a counter or up-down counter. The measurement may have attributes and context. By default, the bound or current context is used.

    (defn get-red-foo [context args]
  (instrument/add! foo-count {:context context
                              :value 1
                              :attributes {:colour :red}})
  (red-foo args))

  • Use steffan-westcott.clj-otel.api.metrics.instrument/record! to record a measurement in a histogram. The measurement may have attributes and context. By default, the bound or current context is used.

    (defn allocate-segment [context size]
  (instrument/record! segment-size {:context context
                                    :value size
                                    :attributes {:partition :public
                                                 :generation :young}})
  (schedule-segment size))

Create and use an instrument to take measurements asynchronously

Counter, up-down counter and gauge instruments support taking measurements asynchronously.

  • Create a 0-arity function that returns a measurement, or a sequence of measurements. The measurement(s) may have attributes.

  • Use steffan-westcott.clj-otel.api.metrics.instrument/instrument to create an instrument of the required type. The second parameter is the function created in the previous step.

    (defn read-temperature []
  (let [temp (get-core-temp)]
    {:value temp
     :attributes {:location :reactor-core}}))

(defonce temperature
  (instrument/instrument {:name "app.temperature"
                          :instrument-type :gauge
                          :measurement-type :long
                          :unit "{degree Celsius}"
                          :description "The operating temperature"}
                         read-temperature))

Add JVM runtime metrics

When running an application with the OpenTelemetry instrumentation agent, the agent automatically adds JVM runtime metrics.

  • Add project dependency :

    deps.edn
    {;; ...
 :deps {com.github.steffan-westcott/clj-otel-instrumentation-runtime-metrics {:mvn/version "0.2.1"}}}

  • Use steffan-westcott.clj-otel.instrumentation.runtime-metrics/register! to add JVM runtime metrics. They are implemented as instruments which take measurements asynchronously.

    (defonce _jvm-reg (runtime-metrics/register!))

Work with HTTP client and server spans

The guides in this section describe semantic conventions support for HTTP client and server spans.

Use Ring middleware for server span support

  • Use Ring middleware steffan-westcott.clj-otel.api.trace.http/wrap-server-span to add HTTP server span support to a Ring handler.

    The enabled support features vary with the selected middleware options. The middleware can be configured to work in applications that run with or without the OpenTelemetry instrumentation agent. It also supports synchronous (1-arity) and asynchronous (3-arity) handlers.

    This is an example using Jetty in an application run with the agent

    (ns example.service
  (:require [ring.adapter.jetty :as jetty]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(defn request-handler [request]
  ;; ...
  )

(def handler
  (-> request-handler
      (trace-http/wrap-server-span {:create-span? false})))

(defonce server
  (jetty/run-jetty #'handler {:port 8080 :join? false}))

    Optionally, to add some HTTP server metrics for applications run without the OpenTelemetry instrumentation agent, add middleware steffan-westcott.clj-otel.api.metrics.http.server/wrap-active-requests and steffan-westcott.clj-otel.api.metrics.http.server/wrap-metrics-by-route.

    This is the same example as above, for an application run without the agent

    (ns example.service
  (:require [ring.adapter.jetty :as jetty]
            [steffan-westcott.clj-otel.api.metrics.http.server :as metrics-http-server]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(defn request-handler [request]
  ;; ...
  )

(def handler
  (-> request-handler
      (metrics-http-server/wrap-metrics-by-route)
      (metrics-http-server/wrap-active-requests)
      (trace-http/wrap-server-span {:create-span? true})))

(defonce server
  (jetty/run-jetty #'handler {:port 8080 :join? false}))

  • If you use middleware that injects data on the matched route into the Ring request map, add middleware steffan-westcott.clj-otel.api.trace.http/wrap-route to add the route data to HTTP server spans for all matched routes.

    This is an example when using Reitit, with Jetty in an application run with the agent

    (ns example.service
  (:require [muuntaja.core :as m]
            [reitit.ring :as ring]
            [ring.adapter.jetty :as jetty]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(defn wrap-reitit-route [handler]
  (trace-http/wrap-route handler
                         (fn [request]
                           (get-in request [:reitit.core/match :template]))))

(defn foo-handler [request]
  ;; ...
  )

(def handler
  (ring/ring-handler (ring/router
                      ["/foo" {:name ::foo :get foo-handler}]
                      {:data {:muuntaja m/instance
                              :middleware [wrap-reitit-route
                                           ;; ... other middleware
                                           ]}})
                     (ring/create-default-handler)

                     ;; Wrap handling of all requests, including those which have no matching route.
                     {:middleware [[trace-http/wrap-server-span {:create-span? false}]]}))

(defonce server
  (jetty/run-jetty #'handler {:port 8080 :join? false}))

    Optionally, to add some HTTP server metrics for applications run without the OpenTelemetry instrumentation agent, add middleware steffan-westcott.clj-otel.api.metrics.http.server/wrap-active-requests and steffan-westcott.clj-otel.api.metrics.http.server/wrap-metrics-by-route.

    This is the same example as above, for an application run without the agent

    (ns example.service
  (:require [muuntaja.core :as m]
            [reitit.ring :as ring]
            [ring.adapter.jetty :as jetty]
            [steffan-westcott.clj-otel.api.metrics.http.server :as metrics-http-server]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(defn wrap-reitit-route [handler]
  (trace-http/wrap-route handler
                         (fn [request]
                           (get-in request [:reitit.core/match :template]))))

(defn foo-handler [request]
  ;; ...
  )

(def handler
  (ring/ring-handler (ring/router
                      ["/foo" {:name ::foo :get foo-handler}]
                      {:data {:muuntaja m/instance
                              :middleware [wrap-reitit-route
                                           metrics-http-server/wrap-metrics-by-route
                                           ;; ... other middleware
                                           ]}})
                     (ring/create-default-handler)

                     ;; Wrap handling of all requests, including those which have no matching route.
                     {:middleware [[trace-http/wrap-server-span {:create-span? true}]
                                   [metrics-http-server/wrap-active-requests]]}))

(defonce server
  (jetty/run-jetty #'handler {:port 8080 :join? false}))

Use Pedestal interceptors for server span support

  • Use steffan-westcott.clj-otel.api.trace.http/server-span-interceptors and steffan-westcott.clj-otel.api.trace.http/route-interceptor to add HTTP server span support to a Pedestal HTTP service.

    The enabled support features vary with the selected interceptor options. The interceptors can be configured to work in applications that run with or without the OpenTelemetry instrumentation agent.

    An example using Jetty in an application run with the agent

    (ns example.service
  (:require [io.pedestal.http :as http]
            [io.pedestal.http.route :as route]
            [io.pedestal.interceptor :as interceptor]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(def routes
  (route/expand-routes ... ))

(defn update-default-interceptors [default-interceptors]
  (map interceptor/interceptor
       (concat (trace-http/server-span-interceptors {:create-span? false})
               default-interceptors
               [(trace-http/route-interceptor)])))

(defn service [service-map]
  (-> service-map
      (http/default-interceptors)
      (update ::http/interceptors update-default-interceptors)
      (http/create-server)))

(def service-map
  {::http/routes routes
   ::http/type   :jetty
   ::http/port   8080
   ::http/join?  false})

(defonce server
  (http/start (service service-map)))

    Optionally, to add some HTTP server metrics for applications run without the OpenTelemetry instrumentation agent, add interceptors steffan-westcott.clj-otel.api.metrics.http.server/active-requests-interceptor and steffan-westcott.clj-otel.api.metrics.http.server/metrics-by-route-interceptors

    This is the same example as above, for an application run without the agent

    (ns example.service
  (:require [io.pedestal.http :as http]
            [io.pedestal.http.route :as route]
            [io.pedestal.interceptor :as interceptor]
            [steffan-westcott.clj-otel.api.metrics.http.server :as metrics-http-server]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]))

(def routes
  (route/expand-routes ... ))

(defn update-default-interceptors [default-interceptors]
  (map interceptor/interceptor
       (concat (trace-http/server-span-interceptors {:create-span? true})
               [(metrics-http-server/active-requests-interceptor)]
               default-interceptors
               [(trace-http/route-interceptor)]
               (metrics-http-server/metrics-by-route-interceptors))))

(defn service [service-map]
  (-> service-map
      (http/default-interceptors)
      (update ::http/interceptors update-default-interceptors)
      (http/create-server)))

(def service-map
  {::http/routes routes
   ::http/type   :jetty
   ::http/port   8080
   ::http/join?  false})

(defonce server
  (http/start (service service-map)))

Manually add route data to a server span

Route data is automatically added to server spans when using the Ring middleware steffan-westcott.clj-otel.api.trace.http/wrap-route or Pedestal interceptor steffan-westcott.clj-otel.api.trace.http/route-interceptor

  • Use steffan-westcott.clj-otel.api.trace.http/add-route-data! to add the matched route to a server span.

    By default, the route data is added to the span in the bound or current context:

    (trace-http/add-route-data! :get "/rooms/:room-id")

    Use the :context option to specify the context containing the span to add the route data to:

    (trace-http/add-route-data! :get "/rooms/:room-id" {:context context})

Manually add HTTP response data to a client span

When running an application with the OpenTelemetry instrumentation agent, the agent automatically adds HTTP response data to client spans for supported clients.

  • Use steffan-westcott.clj-otel.api.trace.http/add-client-span-response-data! to add HTTP response data to a client span. Use this function when working with an HTTP client not supported by the OpenTelemetry instrumentation agent.

    By default, the HTTP response data is added to the span in the bound or current context:

    (trace-http/add-client-span-response-data! response)

    Use the :context option to specify the context containing the span to add the HTTP response data to:

    (trace-http/add-client-span-response-data! response {:context context})

Manually propagate context in an HTTP client request

When running an application with the OpenTelemetry instrumentation agent, the agent automatically propagates the context in HTTP client requests for supported clients.

  • Use steffan-westcott.clj-otel.context/->headers to get headers to merge (inject) with other headers in the HTTP request to be issued for context propagation. Use this function when working with an HTTP client not supported by the OpenTelemetry instrumentation agent.

    By default, the bound or current context is propagated:

    (let [context-headers (context/->headers)
      request' (update request :headers merge context-headers)]
  ;; ...
  )

    Use the :context option to specify the context to be propagated:

    (let [context-headers (context/->headers {:context context})
      request' (update request :headers merge context-headers)]
  ;; ...
  )

Configure and run an application with telemetry

The options below determine what telemetry data is exported from an application as it runs. Select one of these options and follow the linked guide:

Traces and metrics telemetry data are muted in the last option or by setting the autoconfiguration properties otel.traces.exporter and otel.metrics.exporter to none (the defaults are otlp for both properties) when using either of the first two options.

Run with the OpenTelemetry auto-instrumentation agent

  • Download the latest version of the OpenTelemetry instrumentation agent JAR, the file opentelemetry-javaagent.jar from the releases page. The agent JAR includes the SDK and all its dependencies.

  • Configure the agent and SDK using properties and environment variables. See the agent and SDK configuration documentation.

  • When running the application, enable the agent with the -javaagent JVM flag.

For an example application my-app, with deps.edn to export traces only using OTLP over gRPC, use an alias like the following:

deps.edn
{;; ...
 :aliases {
   :otel {:jvm-opts ["-javaagent:path/to/opentelemetry-javaagent.jar"
                     "-Dotel.resource.attributes=service.name=my-app"
                     "-Dotel.traces.exporter=otlp"
                     "-Dotel.metrics.exporter=none"
                     "-Dotel.exporter.otlp.traces.protocol=grpc"]}}}

Run with autoconfigured SDK

  • Add project dependencies:

    • Required: io.opentelemetry/opentelemetry-sdk-extension-autoconfigure for the SDK itself and SDK autoconfiguration.

    • Required: io.opentelemetry/opentelemetry-exporter-??? for any exporters referenced in the configuration. See Java exporter libraries supported by autoconfiguration.

    • Optional: io.opentelemetry.instrumentation/opentelemetry-resources for various resources to be automatically added to telemetry data.

    • Optional: io.opentelemetry.contrib/opentelemetry-aws-resources for various resources describing the AWS execution environment to be automatically added to telemetry data.

    • Optional: io.opentelemetry.contrib/opentelemetry-aws-xray-propagator for text map propagator implementing the AWS X-Ray Trace Header propagation protocol.

    • Optional: io.opentelemetry/opentelemetry-extension-trace-propagators for text map propagators implementing OpenTracing Basic Tracers, Jaeger and B3 propagation protocols.

    • Optional: io.grpc/grpc-netty-shaded, io.grpc/grpc-protobuf and io.grpc/grpc-stub to use Netty for gRPC transport rather than the default OkHttp (see example below). This is not needed if gRPC is not used by any exporters or the application.

  • Configure the SDK using properties and environment variables.

    • Include the JVM option "-Dotel.java.global-autoconfigure.enabled=true" or environment variable setting OTEL_JAVA_GLOBAL_AUTOCONFIGURE_ENABLED=true

    • See SDK autoconfigure configuration documentation for details on all autoconfiguration options.

For an example application my-app, with deps.edn to export traces only using OTLP over gRPC with Netty transport, use an alias like the following:

deps.edn
{;; ...
 :aliases {
   :otel {:jvm-opts ["-Dotel.resource.attributes=service.name=my-app"
                     "-Dotel.java.global-autoconfigure.enabled=true"
                     "-Dotel.traces.exporter=otlp"
                     "-Dotel.metrics.exporter=none"
                     "-Dotel.exporter.otlp.traces.protocol=grpc"]
          :extra-deps {io.opentelemetry/opentelemetry-sdk-extension-autoconfigure {:mvn/version "1.26.0-alpha"}
                       io.opentelemetry/opentelemetry-exporter-otlp               {:mvn/version "1.26.0"}
                       io.opentelemetry.instrumentation/opentelemetry-resources   {:mvn/version "1.26.0-alpha"}
                       io.grpc/grpc-netty-shaded                                  {:mvn/version "1.55.1"}
                       io.grpc/grpc-protobuf                                      {:mvn/version "1.55.1"}
                       io.grpc/grpc-stub                                          {:mvn/version "1.55.1"}}}}}

Run with programmatically configured SDK

  • Add project dependencies:

    • Required: com.github.steffan-westcott/clj-otel-sdk for the SDK itself and a Clojure wrapper of SDK configuration

    • Required: com.github.steffan-westcott/clj-otel-exporter-??? for Clojure wrapped versions of any exporters referenced in the configuration. See Clojure wrapped versions of exporters supported by autoconfiguration.

    • Optional: com.github.steffan-westcott/clj-otel-sdk-extension-resources for Clojure wrapped versions of various resources to add to telemetry data.

    • Optional: com.github.steffan-westcott/clj-otel-contrib-aws-resources for Clojure wrapped versions of resources describing the AWS execution environment.

    • Optional: com.github.steffan-westcott/clj-otel-contrib-aws-xray-propagator for Clojure wrapped text map propagator implementing the AWS X-Ray Trace Header propagation protocol.

    • Optional: com.github.steffan-westcott/clj-otel-extension-trace-propagators for Clojure wrapped text map propagators implementing OpenTracing Basic Tracers, Jaeger and B3 propagation protocols.

    • Optional: io.grpc/grpc-netty-shaded, io.grpc/grpc-protobuf and io.grpc/grpc-stub to use Netty for gRPC transport rather than the default OkHttp (see example below). This is not needed if gRPC is not used by any exporters or the application.

  • At application start, use steffan-westcott.clj-otel.sdk.otel-sdk/init-otel-sdk! to configure and set an OpenTelemetry SDK instance as the global OpenTelemetry instance.

  • At application end, use steffan-westcott.clj-otel.sdk.otel-sdk/close-otel-sdk! to close down activities of the SDK instance.

For an example application my-app, with deps.edn to export traces only using OTLP over gRPC with Netty transport, use deps like the following:

deps.edn
{;; ...
 :deps {com.github.steffan-westcott/clj-otel-exporter-otlp            {:mvn/version "0.2.1"}
        com.github.steffan-westcott/clj-otel-sdk-extension-resources  {:mvn/version "0.2.1"}
        com.github.steffan-westcott/clj-otel-sdk                      {:mvn/version "0.2.1"}
        io.grpc/grpc-netty-shaded                                     {:mvn/version "1.55.1"}
        io.grpc/grpc-protobuf                                         {:mvn/version "1.55.1"}
        io.grpc/grpc-stub                                             {:mvn/version "1.55.1"}}}

To configure the SDK at start and close down at end, the application could have functions like the following:

example/app.clj
(ns example.app
  (:require [steffan-westcott.clj-otel.exporter.otlp-grpc-trace :as otlp-grpc-trace]
            [steffan-westcott.clj-otel.resource.resources :as res]
            [steffan-westcott.clj-otel.sdk.otel-sdk :as sdk]))

(defn init-otel! []
  (sdk/init-otel-sdk!
    "my-app"
    {:resources [(res/host-resource)
                 (res/os-resource)
                 (res/process-resource)
                 (res/process-runtime-resource)]
     :tracer-provider
       {:span-processors
         [{:exporters [(otlp-grpc-trace/span-exporter)]}]}}))

(defn close-otel! []
  (sdk/close-otel-sdk!))

Run without agent or SDK

There are no steps to add dependencies or otherwise configure the application to run without the agent or SDK.

An application run without the OpenTelemetry instrumentation agent or SDK will not export any telemetry data. Usage of the OpenTelemetry API (manual instrumentation) in the application will invoke no-op implementations.

Use the OpenTelemetry Collector

The OpenTelemetry Collector is used to manage telemetry data, as an alternative to applications exporting data directly to telemetry backends.

Deploy the OpenTelemetry Collector

See the getting started documentation.

Configure the OpenTelemetry Collector

See the configuration documentation.

This example Collector configuration has a traces pipeline where:

  • Trace data are received by the Collector as OTLP over gRPC

  • Memory usage in the Collector process is limited

  • Traces are batched in the Collector prior to export to the backend

  • Traces are exported by the Collector as OTLP over gRPC to the Jaeger backend on host jaeger

otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 750
  batch:

exporters:
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ otlp ]
❮TutorialAPI & Reference❯

Can you improve this documentation?Edit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close