wiretap

A Clojure library for adding generic trace support without having to modify code.

wiretap | ˈwʌɪətap |

[noun]

A concealed device connected to a telephone or other communications system that allows a third party to listen or record conversations.

[verb]

To install or to use such a device.

This library provides a small set of tools that help you to observe the execution of functions and multimethods. It is designed to be used in (dev) environments where you want to gain insights without having to modify/annotate code.

Releases

As a git dep:

io.github.beoliver/wiretap {:git/sha "12a3640"}

As a Maven dep:

io.github.beoliver/wiretap {:mvn/version "0.0.17"}

Quick Start

The simplest way to use wiretap is with the record namespace:

(require '[wiretap.record :as rec])

;; Start recording all functions in a namespace
(def recorder (rec/start! {:globs ["my-app.core"]}))

;; Run your code
(my-app.core/some-function 42)

;; Play back the formatted trace
(rec/playback recorder)

;; Access raw events for analysis
(rec/events recorder)

;; Clean up when done
(rec/stop! recorder)

Recording with glob patterns

Glob patterns use * and ** wildcards to match namespaces:

;; * matches a single namespace segment
(rec/start! {:globs ["my-app.*.utils"]})
;; Matches: my-app.core.utils, my-app.web.utils
;; Does NOT match: my-app.utils, my-app.core.web.utils

;; ** matches zero or more segments
(rec/start! {:globs ["my-app.**"]})
;; Matches: my-app.core, my-app.core.utils, my-app.web.handlers.auth

;; Record multiple patterns
(rec/start! {:globs ["my-app.core" "my-app.util.*"]})

;; Record specific functions
(rec/start! {:vars [#'my-app.core/foo #'other.ns/bar]})

;; Combine both approaches
(rec/start! {:globs ["my-app.**"]
             :vars [#'other.ns/helper]})

Working with recordings

(def recorder (rec/start! {:globs ["my-app.**"]}))

;; Run some code to generate events
(my-app.core/process-data {:user-id 123})

;; Display formatted trace output
(rec/playback recorder)
;; TRACE t123: (my-app.core/process-data {:user-id 123})
;; TRACE t124: | (my-app.db/fetch-user 123)
;; TRACE t124: | => {:id 123 :name "Alice"}
;; TRACE t123: => {:processed true}

;; Access events directly for custom analysis
(def events (rec/events recorder))

;; Filter for specific functions

(->> (rec/events recorder)
     (into []
       (comp (filter :post?)
             (filter #(= 'fetch-user (:name %)))
             (map :result))))

;; Clear recorded events (keeps recording)
(rec/wipe! recorder)

;; Stop recording and restore original functions
(rec/stop! recorder)

How it Works

Wiretap works by wrapping function calls. Any var whose value is an instance of Fn or MultiFn (created via fn or defmulti) can be wiretapped.

As a user of the library, you provide a function that will be called before and/or after any wiretapped function or method is invoked. This function can be used to perform any side effecting operations - for example swapping values in an atom, or simply calling println. This custom function is passed a context map that contains information about the invocation of the var. For more information see wiretap context.

A Simple Example

(ns user)
(defn foo [x] (inc x))
(defn bar [x] (foo (dec x)))

user=> (require '[wiretap.wiretap :as wiretap])
nil
user=> (wiretap/install! #(println (if (:pre? %) ">" "<") (:name %)) [#'foo #'bar])
(#'user/foo #'user/bar)
user=> (+ 10 (bar 1))
> bar
> foo
< foo
< bar
11

Supported Types

Depending on the type of the var value, different strategies are used to wrap the invocation of the var. The following table shows the supported types and the corresponding wrapping strategy.

Instance	wiretap
Fn	Wraps invocation of the function
MultiFn	Wraps invocation of the selected method determined by the dispatch-value

Wiretap Context

Depending on whether the user provided function is called before or after the invocation of the wiretapped var, the context map will contain different information. The following table shows the keys that will be present in the context map and when they will be present.

Key	When	Value
:id	pre/post	Uniquely identifies the call. Same value for pre and post calls.
:name	pre/post	A symbol. Taken from the meta of the var.
:ns	pre/post	A namespace. Taken from the meta of the var.
:function	pre/post	The value that will be applied to the value of :args.
:thread	pre/post	The name of the thread.
:stack	pre/post	The current stacktrace.
:depth	pre/post	Number of wiretapped function calls on the stack.
:args	pre/post	The seq of args that value of :function will be applied to.
:start	pre/post	Nanoseconds since some fixed but arbitrary origin time.
:parent	pre/post	The context of the previous wiretapped function on the stack.
:pre?	pre	true
:post?	post	true
:stop	post	Nanoseconds since some fixed but arbitrary origin time.
:result	post	The result of applying the value of :function to :args.
:error	post	Any exception caught during computation of the result.

If the wiretapped var is a multimethod then the following information will also be present.

Key	When	Value
:multimethod?	pre/post	true
:dispatch-val	pre/post	The dispatch value used to select the method.

API Overview

wiretap.record - High-level recording API

For most use cases - simple recording and playback:

• (start! {:globs [...] :vars [...]}) - Start recording matching functions
• (events recorder) - Get vector of recorded events
• (playback recorder) - Display formatted trace to console
• (wipe! recorder) - Clear recorded events (keeps recording)
• (stop! recorder) - Stop recording and restore original functions

Returns a "recorder" map with :vars and :events keys.

wiretap.wiretap - Low-level wiretap API

For custom tracers and advanced use cases:

• (install! tracer-fn vars) - Install custom tracer on vars
• (install-pre! tracer-fn vars) - Tracer called only before invocation
• (install-post! tracer-fn vars) - Tracer called only after invocation
• (uninstall! vars) - Remove wiretap from vars
• (uninstall!) - Remove all wiretaps

Your tracer function receives the context map on each call.

wiretap.tools - Utility functions

Helper functions for working with vars and traces:

• (glob-regex pattern) - Convert glob string to regex Pattern
• (ns-matches regex) - Find namespace symbols matching regex
• (ns-matches-vars regex) - Find vars in matching namespaces
• (globs-vars glob-patterns) - Find vars matching glob patterns
• (ns-vars & namespaces) - Get all vars from namespaces
• (wiretapped? var) - Check if var is currently wiretapped
• (display-trace events) - Format and print event sequence

Documentation hosted here

Examples

Multimethods

Assume that we have a multimethod

(defmulti m1 :name)

(defmethod m1 :foo [x] {:the-foo x})
(defmethod m1 :bar [x] {:the-bar x})

We can wiretap this as follows

user=> (wiretap/install!
        #(when (:pre? %)
          (println (select-keys % [:name :dispatch-val :args]))) [#'m1])
(#'user/m1)
user=> (m1 {:name :foo})
{:name m1, :dispatch-val :foo, :args ({:name :foo})} ;; printed line
{:the-foo {:name :foo}}
user=> (m1 {:name :bar})
{:name m1, :dispatch-val :bar, :args ({:name :bar})}  ;; printed line
{:the-bar {:name :bar}}

However, if we add a new method then it will not be wiretapped.

user=> (defmethod m1 :baz [x] {:the-baz x})
#multifn[m1 0x2f3911be]
user=> (m1 {:name :baz})
{:the-baz {:name :baz}}

The methods that were wiretapped, remain wiretapped.

user=> (m1 {:name :foo})
{:name m1, :dispatch-val :foo, :args ({:name :foo})} ;; printed line
{:the-foo {:name :foo}}

When uninstalling, the wiretapped methods are replaced with the original ones.

user=> (wiretap/uninstall! [#'m1])
(#'user/m1)
user=> (m1 {:name :foo})
{:the-foo {:name :foo}}
user=> (m1 {:name :bar})
{:the-bar {:name :bar}}
user=> (m1 {:name :baz})
{:the-baz {:name :baz}}

Writing a tools.trace clone

Assume that we have the following namespace definitions...

(ns user)

(defn simple [x] (inc x))

(defn call-f [f x] (f x))

(defn pass-simple [x] (call-f simple x))

To show how wiretap events can be used - we will generate traces similar to those of the clojure/tools.trace library. All we need to do is write a function that can take a wiretap context map and perform some io (call to println).

(defn ^:wiretap.wiretap/exclude my-trace
  [trace-id-atom {:keys [id pre? depth name ns args result] :as ctx}]
  (let [trace-id (if pre? (gensym "t") (get @trace-id-atom id))
        trace-indent (apply str (take depth (repeat "| ")))
        trace-value (if pre?
                      (str trace-indent (pr-str (cons (symbol (ns-resolve ns name)) args)))
                      (str trace-indent "=> " (pr-str result)))]
    (if pre?
      (swap! trace-id-atom assoc id trace-id)
      (swap! trace-id-atom dissoc id))
    (println (str "TRACE" (str " " trace-id) ": " trace-value))))

To make things interesting - we will persist all of the contexts and then run our trace function on the data. Repeatable traces!

user=> (require '[wiretap.tools :as tools])
nil
user=> (def history (atom []))
#'user/history
user=> (wiretap/install! #(swap! history conj %) (tools/ns-vars *ns*))
(#'user/pass-simple #'user/simple #'user/call-f)
user=> (pass-simple 1)
2
user=> (count @history)
6
user=> (run! (partial my-trace (atom {})) @history)
TRACE t8018: (user/pass-simple 1)
TRACE t8019: | (user/call-f #function[clojure.lang.AFunction/1] 1)
TRACE t8020: | | (user/simple 1)
TRACE t8020: | | => 2
TRACE t8019: | => 2
TRACE t8018: => 2
nil

Or using the record namespace:

user=> (require '[wiretap.record :as rec])
nil
user=> (def recorder (rec/start! {:globs ["user"]}))
#'user/recorder
user=> (pass-simple 1)
2
user=> (rec/playback recorder)
TRACE t8021: (user/pass-simple 1)
TRACE t8022: | (user/call-f #function[clojure.lang.AFunction/1] 1)
TRACE t8023: | | (user/simple 1)
TRACE t8023: | | => 2
TRACE t8022: | => 2
TRACE t8021: => 2
nil

Inferring specs

Now that we have a history of events, we can perform other operations on them! In the previous example we called pass-simple passing the value 1. Let's use the spec-provider library to infer some specs from the trace.

(require '[spec-provider.provider :as sp])

(defn result-spec [history var-obj]
  (let [var-ns (:ns (meta var-obj))
        var-name (:name (meta var-obj))
        examples (->> history
                      (filter (fn [{:keys [post? error ns name]}]
                                (and post?
                                     (nil? error) ;; ignore results if error thrown
                                     (= ns var-ns)
                                     (= name var-name))))
                      (map :result))]
    (sp/pprint-specs
     (sp/infer-specs (set examples) (keyword (name (ns-name var-ns))
                                             (name var-name)))
     var-ns 'spec)))

We can now use the function to infer the spec of the return value for a function - even if we never called it directly.

=> (result-spec @history #'simple)
(spec/def ::simple integer?)
=> (call-f simple 2.0)
3.0
=> (result-spec @history #'simple)
(spec/def ::simple (spec/or :double double? :integer integer?))

Or using the record namespace:

=> (result-spec (rec/events recorder) #'simple)
(spec/def ::simple integer?)

Related

• https://github.com/clojure/tools.trace
• https://github.com/circleci/bond
• https://github.com/alexanderjamesking/spy
• https://github.com/technomancy/robert-hooke

Development

clj -X:test

Test in the REPL

clj -Sdeps '{:deps {wiretap/wiretap {:git/url "https://github.com/beoliver/wiretap/" :git/sha "12a3640"}}}' -e "(require '[wiretap.wiretap :as wiretap] '[wiretap.tools :as wiretap-tools])" -r

Checking out: https://github.com/beoliver/wiretap/ at 12a3640994ff8241cdd38995d16c481bcf143c53
WARNING: Implicit use of clojure.main with options is deprecated, use -M
user=> (def foo (fn [x] (+ x x)))
#'user/foo
user=> (wiretap/install! #(when (:post? %) (clojure.pprint/pprint %)) [#'foo])
(#'user/foo)
user=> (foo 10)
{:args (10),
 :parent nil,
 :ns #object[clojure.lang.Namespace 0x309028af "user"],
 :name foo,
 :start 298028669941875,
 :function #object[user$foo 0x44841b43 "user$foo@44841b43"],
 :stop 298028670197791,
 :result 20,
 :thread "main",
 :post? true,
 :id "7bd32775-d675-48ab-8d42-c56924ed7ee3",
 :stack
 [[java.lang.Thread getStackTrace "Thread.java" 1602],
  [wiretap.wiretap$wiretap_var_BANG_$wiretapped__149 doInvoke "wiretap.clj" 17],
  [clojure.lang.RestFn applyTo "RestFn.java" 137],
  [clojure.lang.AFunction$1 doInvoke "AFunction.java" 31],
  [clojure.lang.RestFn invoke "RestFn.java" 408],
  [user$eval223 invokeStatic "NO_SOURCE_FILE" 1],
  [user$eval223 invoke "NO_SOURCE_FILE" 1],
  [clojure.lang.Compiler eval "Compiler.java" 7194],
  [clojure.lang.Compiler eval "Compiler.java" 7149],
  [clojure.core$eval invokeStatic "core.clj" 3215],
  [clojure.core$eval invoke "core.clj" 3211],
  [clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437],
  [clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437],
  [clojure.main$repl$fn__9215 invoke "main.clj" 458],
  [clojure.main$repl invokeStatic "main.clj" 458],
  [clojure.main$repl_opt invokeStatic "main.clj" 522],
  [clojure.main$repl_opt invoke "main.clj" 518],
  [clojure.main$main invokeStatic "main.clj" 664],
  [clojure.main$main doInvoke "main.clj" 616],
  [clojure.lang.RestFn applyTo "RestFn.java" 137],
  [clojure.lang.Var applyTo "Var.java" 705],
  [clojure.main main "main.java" 40]],
 :depth 0}
20
user=> (wiretap/uninstall!)
(#'user/foo)
user=> (foo 20)
40