Tutorial

Tutorial

In this tutorial, you will add trace telemetry to a Clojure HTTP server application and view the traces in Jaeger, a telemetry backend.

First, you will add automatic instrumentation to get telemetry without changing any application code. You will then add manual instrumentation to enrich the telemetry data and gain further insight into the application behaviour.

The tutorial walks through the creation and instrumentation of the application from scratch. The application source code is in the tutorial directory of this repository, showing before and after instrumentation.

The tutorial assumes a Unix-like OS equipped with Docker and a Clojure development environment. It also assumes you are comfortable working in a Clojure REPL and are familiar with Ring-based HTTP server development.

Table of Contents

The counter-service application
Start telemetry backend
Add automatic instrumentation
Add manual instrumentation

The `counter-service` application

The counter-service application is a simple Jetty HTTP service using Ring that maintains an integer counter.

PUT /reset?n=3 resets the counter, here to the value 3
GET /count returns the current counter value
POST /inc increments the counter by 1

The application has no instrumentation initially.

In your favourite development environment, create a project with these two files:

./deps.edn

{:paths ["src"]

 :deps  {org.clojure/clojure     {:mvn/version "1.11.2"}
         ring/ring-jetty-adapter {:mvn/version "1.12.2"}}}

./src/tutorial/counter_service.clj

(ns tutorial.counter-service
  (:require [ring.adapter.jetty :as jetty]
            [ring.middleware.params :as params]
            [ring.util.response :as response]))

(defonce counter (atom 0))

(defn wrap-exception [handler]
  (fn [request]
    (try
      (handler request)
      (catch Throwable e
        (let [resp (response/response (ex-message e))]
          (response/status resp 500))))))

(defn reset-count-handler [{:keys [query-params]}]
  (let [n (Integer/parseInt (get query-params "n"))]
    (reset! counter n)
    (response/status 204)))

(defn get-count-handler []
  (let [n @counter]
    (response/response (str n))))

(defn inc-count-handler []
  (swap! counter inc)
  (response/status 204))

(defn handler [{:keys [request-method uri] :as request}]
  (case [request-method uri]
    [:put "/reset"] (reset-count-handler request)
    [:get "/count"] (get-count-handler)
    [:post "/inc"] (inc-count-handler)
    (response/not-found "Not found")))

(def service
  (-> handler
      params/wrap-params
      wrap-exception))

(defonce server
  (jetty/run-jetty #'service {:port 8080 :join? false}))

In a REPL load the namespace tutorial.counter-service to start the server.

With counter-service running, use curl to try it out:

curl -X PUT "http://localhost:8080/reset?n=3"
curl -X GET "http://localhost:8080/count"
# 3
curl -X POST "http://localhost:8080/inc"
curl -X GET "http://localhost:8080/count"
# 4

Start telemetry backend

You will now start a telemetry backend that will receive and display telemetry trace data exported by the application.

Enter the following command to start a Jaeger instance in a container using Docker:

docker run --rm                     \
           -p 16686:16686           \
           -p 4318:4318             \
           jaegertracing/all-in-one \
           --collector.otlp.enabled=true

Once Jaeger has started, verify you can access the user interface by navigating to http://localhost:16686/search in a web browser.

There is no telemetry data to view yet as the application is not instrumented.

When finished, use CTRL+C to tear down the container.

Add automatic instrumentation

The application uses Jetty, which is supported by the OpenTelemetry instrumentation agent.

Download the file opentelemetry-javaagent.jar from the OpenTelemetry instrumentation agent releases page. Ensure the file is placed in the project root directory (the same directory as deps.edn)

Add an alias :otel:

./deps.edn

{:paths   ["src"]

 :deps    {org.clojure/clojure     {:mvn/version "1.11.2"}
           ring/ring-jetty-adapter {:mvn/version "1.12.2"}}

 :aliases {:otel {:jvm-opts ["-javaagent:opentelemetry-javaagent.jar"
                             "-Dotel.resource.attributes=service.name=counter-service"
                             "-Dotel.metrics.exporter=none"
                             "-Dotel.logs.exporter=none"]}}}

Restart the REPL with the alias :otel enabled and reload the tutorial.counter-service namespace to run the server with automatic instrumentation.

As the application handles each request, now a trace is exported to Jaeger.

Exercise the server by sending a couple of requests:

curl -X PUT "http://localhost:8080/reset?n=7"
curl -X GET "http://localhost:8080/count"
# 7

In a web browser navigate to the Jaeger search page at http://localhost:16686/search. In the Search options, select counter-service in the Service selector and click the Find Traces button.

You will see a trace corresponding to each request handled by the server.
Click on the trace for HTTP GET to view it, then click on the single span in the trace to expand its details.

The span’s Tags attributes describe this as a server span for an HTTP GET request for target /count with an HTTP response code of 200. The span’s Process attributes describe the instrumented application and its environment, such as the service name, runtime JVM, process, OS and host.

Add manual instrumentation

You will now enrich the telemetry detail by adding manual instrumentation in addition to the existing automatic instrumentation.

Add a dependency com.github.steffan-westcott/clj-otel-api:

./deps.edn

{:paths   ["src"]

 :deps    {org.clojure/clojure                      {:mvn/version "1.11.2"}
           ring/ring-jetty-adapter                  {:mvn/version "1.12.2"}
           com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.6"}}

 :aliases {:otel {:jvm-opts ["-javaagent:opentelemetry-javaagent.jar"
                             "-Dotel.resource.attributes=service.name=counter-service"
                             "-Dotel.metrics.exporter=none"]}}}

Restart the REPL (again with the alias :otel enabled) to pick up the new dependency.

Update the Ring service definition to add server span support:

./src/tutorial/counter_service.clj

(ns tutorial.counter-service
  (:require [ring.adapter.jetty :as jetty]
            [ring.middleware.params :as params]
            [ring.util.response :as response]
            [steffan-westcott.clj-otel.api.trace.http :as trace-http]
            [steffan-westcott.clj-otel.api.trace.span :as span]))

;; ...

(def service
  (-> handler
      params/wrap-params
      wrap-exception
      trace-http/wrap-server-span))

Update the get-count-handler function to add an attribute service.counter/count to the existing server span as follows:

(defn get-count-handler []
  (let [n @counter]
    (span/add-span-data! {:attributes {:service.counter/count n}})
    (response/response (str n))))

Reload the namespace in the REPL and issue some more requests:

curl -X PUT "http://localhost:8080/reset?n=5"
curl -X GET "http://localhost:8080/count"
# 5

In the Jaeger UI, navigate back to the search page and click the Find Traces button again to display the new traces. Click the most recent HTTP GET trace and expand the Tags of the single span.

You will note that the attribute service.counter.count you added in the code appears in the exported span with value 5.

Update the inc-count-handler function to add a new span that wraps part of the function body:

(defn inc-count-handler []
  (span/with-span! "Incrementing counter"
    (swap! counter inc))
  (response/status 204))

Reload the namespace once more and exercise the function:
```
curl -X POST "http://localhost:8080/inc"
```
Find the HTTP POST trace on the Jaeger search page (remember to refresh with the Find Traces button) and click to view its details.

Notice the trace has two spans; a root server span named HTTP POST and a child internal span named Incrementing counter that you added to the code.

Update the function wrap-exception to add detail about any caught exceptions to the server span:

(defn wrap-exception [handler]
  (fn [request]
    (try
      (handler request)
      (catch Throwable e
        (span/add-exception! e {:escaping? false})
        (let [resp (response/response (ex-message e))]
          (response/status resp 500))))))

Reload the namespace, then issue a malformed request to cause an exception:

curl -X PUT "http://localhost:8080/reset?bogus=1"
# Cannot parse null string

Find the most recent HTTP PUT trace on the Jaeger search page. You will see that the server span has a Logs event added by span/add-exception!. The event attributes include a Clojure triage and a stack trace of the caught exception.

The counter-service application

Start telemetry backend

Add automatic instrumentation

Add manual instrumentation

The `counter-service` application