Liking cljdoc? Tell your friends :D

Paclo

cljdoc Clojars Project

Paclo is a Clojure library for packet capture (PCAP) I/O, BPF filtering, and data-first packet processing. It provides a small public API plus optional decode hooks for DNS/TLS workflows.

If cljdoc or Clojars returns 404 right after publishing, indexing can take several minutes.

Project status

The install coordinates below select the verified v1.4.0 Clojars release. Features identified as v1.4.0 are included in that published artifact.

Repository v1.4.0 captures expose native DLT :linktype. Raw reads support non-Ethernet PCAPs; full/flow decode supports Ethernet/VLAN (DLT 1), Linux SLL (113), SLL2 (276), Raw IP (12), NULL (0) and LOOP (108) IPv4/IPv6 on Linux/macOS. Other explicit types produce :unsupported-linktype diagnostics. Standalone packet-xf without the key retains its Ethernet assumption. P08-B2 extends core/write-pcap! with {:linktype native-DLT :snaplen positive-integer}. Two arguments default to Ethernet (1) and snaplen 65536. Explicit packet DLTs must match the writer; non-Ethernet maps now require a matching option. Bytes without linktype metadata trust the caller's selected DLT; no bytes are sniffed or converted. See the cooked capture examples and writer contract.

For ordered PCAPNG sections/interfaces/options and exact timestamps, use the separate paclo.pcapng APIs (v1.4.0 P08-E2/E3). It owns a bounded synchronous FileChannel source that you reduce once with with-open. pcapng/write-pcapng! consumes ordered events synchronously and publishes a new file with exclusive atomic rename on supported Linux/macOS file systems. The default preserves known metadata; lossy drops and authored input require explicit modes. See the executable writer examples.

Decode SLL and SLL2 with the same transducer; each retains its own link header:

(require '[paclo.core :as core])

(mapv (fn [path]
        (let [packet (first (core/reduce-packets
                            {:path path :max 1 :decode? true :extensions []} conj []))]
          {:linktype (:linktype packet)
           :link (:type (:decoded packet))
           :ip (get-in packet [:decoded :l3 :type])
           :port (get-in packet [:decoded :l3 :l4 :dst-port])}))
      ["test/resources/cooked/sll.pcap" "test/resources/cooked/sll2.pcap"])

TCP/UDP/ICMP, fragments, IPv6 extension headers and explicit DNS/TLS extensions reuse the existing IP/L4 parsers. Flow projection retains numeric fields and works with paclo.flow aggregates. Cooked addresses describe one link-layer address, not an Ethernet MAC pair; SLL2 :interface-index is separate from a PCAPNG interface ID.

Raw IP begins at byte zero, with no link header. The version nibble selects the same IPv4/IPv6 parsers; full decode wraps them in {:type :raw-ip :l3 ...}. Savefile LINKTYPE_RAW 101 is translated by libpcap to native DLT 12 on the measured Linux/macOS matrix. Supply native DLT metadata/options, not 101; unverified platforms remain unsupported. No Ethernet or SLL fields are synthesized.

(require '[paclo.core :as core])

(let [packets (core/reduce-packets {:path "test/resources/raw-ip/raw.pcap"
                                  :max 8 :decode? true :extensions []} conj [])]
  {:linktypes (set (map :linktype packets))
   :links (set (map #(get-in % [:decoded :type]) packets))
   :versions (set (map #(get-in % [:decoded :l3 :version]) packets))})

NULL/LOOP have a four-byte address-family header and IP at offset 4. NULL accepts a unique known AF under either byte order; LOOP requires big endian. AF 2 is IPv4, 24/28/30 are IPv6; unknown AFs (including 10) and AF/version mismatches fail explicitly. Full uses {:type :null :address-family 2 :l3 ...} or :type :loop; flow keeps the usual numeric fields. Raw bytes are preserved. Measured Linux x64/ARM64 and macOS ARM64 libraries leave these headers unchanged. Other OS mappings remain unsupported. Packet maps omit original container byte order, so decoding cannot verify container/header consistency or reproduce its encoding. See the NULL/LOOP contract.

(require '[paclo.core :as core])

(mapv (fn [path]
        (let [packet (first (core/reduce-packets
                            {:path path :max 1 :decode? true :extensions []} conj []))]
          {:linktype (:linktype packet)
           :link (:type (:decoded packet))
           :af (get-in packet [:decoded :address-family])
           :ip (get-in packet [:decoded :l3 :version])}))
      ["test/resources/null-loop/null-header-be-file-le.pcap"
       "test/resources/null-loop/loop-header-be-file-be.pcapng"])

Write and read back Linux cooked packets with their native DLT (113):

(require '[paclo.core :as core])

(let [out (java.io.File/createTempFile "paclo-sll-" ".pcap")
      source (core/reduce-packets {:path "test/resources/linktype-sll.pcap"} conj [])
      snapshot #(update (select-keys % [:bytes :ts-sec :ts-usec :caplen :len :linktype])
                        :bytes vec)]
  (try
    (core/write-pcap! source (.getAbsolutePath out) {:linktype 113 :snaplen 65536})
    (let [saved (core/reduce-packets {:path (.getAbsolutePath out)} conj [])]
      {:count (count saved)
       :linktypes (mapv :linktype saved)
       :preserved? (= (mapv snapshot source) (mapv snapshot saved))})
    (finally (.delete out))))

Output is always single-DLT Classic PCAP with microsecond timestamps, regardless of suffix. Packet order, bytes, timestamps and captured/original lengths are preserved; captured length must not exceed snaplen. PCAPNG interface metadata, comments, sections and sub-microsecond precision cannot be preserved. This is not byte-for-byte source-file reproduction. Failure can leave an empty file or valid prefix; see write options and errors.

Install

deps.edn (recommended: Clojars)

{:deps
 {org.clojars.nanto/paclo {:mvn/version "1.4.0"}}}

deps.edn (Git tag)

{:deps
 {org.clojars.nanto/paclo
  {:git/url "https://github.com/nantes-rfli/paclo.git"
   :git/tag "v1.4.0"}}}

If your environment requires Java class prep for git dependencies, run once in your consumer project:

clojure -X:deps prep

Quick start

(require '[paclo.core :as core])

;; Read the first two decoded packets from the bundled offline fixture.
(with-open [capture (core/start-capture
                     {:path "test/resources/dns-sample.pcap" :decode? true})]
  (into [] (comp (take 2)
                 (map #(select-keys % [:ts-sec :ts-usec :decoded :decode-error])))
        (core/capture-packets capture)))

For offline or live capture, use start-capture with with-open when taking a prefix or stopping early. take on the lazy packets sequence alone does not stop its producer; a full blocking queue can retain the native handle indefinitely, even past capture time limits. The usage guide explains ownership and safe alternatives.

For bounded flow analysis, select numeric flow projection and reuse the public paclo.flow reducer. IPv4 addresses are unsigned integers; IPv6 addresses are two-long vectors. Exact byte totals distinguish captured length from wire length.

(require '[paclo.flow :as flow])

(let [rf (flow/aggregate-rf {:direction :bidir :max-flows 10000})
      result (core/reduce-packets
              {:path "test/resources/dns-sample.pcap" :decode-mode :flow}
              rf (rf))]
  (flow/top-n result {:metric :captured-bytes :n 10}))

flow/key defines canonical numeric identity; full decode's legacy [:decoded :l3 :flow-key] remains a string-address display descriptor. Use flow projection for analysis and full decode for protocol inspection. See the flow analysis contract and composition examples for ordering, invalid inputs and cardinality limits.

For transformations outside capture APIs, compose the same numeric projection with standard transducers:

(into []
      (comp (core/packet-xf {:decode-mode :flow})
            (filter #(= 17 (:protocol %))) ; UDP
            (map #(select-keys %
                               [:src-ip :dst-ip :src-port :dst-port])))
      (core/packets {:path "test/resources/dns-sample.pcap"}))

Use either :decode? true for full decoding with decode extensions, or :decode-mode :flow for a flat flow projection (not both). Capture APIs apply :xform after the built-in packet transformation.

In v1.4.0 (P03), full decode accepts an ordered :extensions vector of functions. Omission keeps dynamic global hooks; [] disables them, and explicit selection runs only its functions. Public DNS/TLS annotations and independent fan-out examples are in decode extensions.

For synchronous processing with data-only execution statistics:

(core/reduce-packets-report
 {:path "large.pcap"
  :decode-mode :flow
  :max Long/MAX_VALUE
  :max-time-ms 3600000}
 (fn [counts flow]
   (update counts
           (select-keys flow
                        [:src-ip :dst-ip :protocol :src-port :dst-port])
           (fnil inc 0)))
 {})
;; => {:result {...}
;;     :stats {:schema-version 1
;;             :state :closed
;;             :stop-reason :eof
;;             ...}}

For long-running or externally stopped capture, own a managed capture with with-open. Packet delivery is single-consumer. A quiet device otherwise stops after the default 3-second idle limit, so this example sets :idle-max-ms explicitly to one hour, matching :max-time-ms:

(with-open [capture
            (core/start-capture
             {:device "en0"
              :filter [:and :udp [:port 53]]
              :queue-cap 4096
              :queue-mode :dropping
              :max Long/MAX_VALUE
              :max-time-ms 3600000
              :idle-max-ms 3600000})]
  (run! #(prn (select-keys % [:ts-sec :ts-usec :caplen]))
        (core/capture-packets capture))
  (core/capture-stats capture))

To feed two independent consumers, compose a managed capture with bounded fan-out. This example runs offline with the bundled fixture. Each branch is single-consumer and must be owned with with-open:

(require '[paclo.stream :as stream])

(with-open [capture
            (core/start-capture
             {:path "test/resources/dns-sample.pcap"})
            fanout
            (stream/fan-out
             (core/capture-packets capture)
             {:archive {:buffer-mode :blocking :buffer-cap 16}
              :metrics {:buffer-mode :dropping :buffer-cap 16}}
             {:cancel! #(core/stop-capture! capture)})
            archive (stream/branch fanout :archive)
            metrics (stream/branch fanout :metrics)]
  (let [archive-task (future (reduce (fn [n _] (inc n)) 0 archive))
        metrics-task (future (reduce (fn [n _] (inc n)) 0 metrics))]
    {:archive-count @archive-task
     :metrics-count @metrics-task
     :capture (core/capture-stats capture)
     :stream (stream/stats fanout)}))

Blocking buffers can backpressure other branches; dropping buffers isolate slow consumers by discarding new values when full. Both remain bounded. For full behavior, shutdown rules, and stop defaults, see the usage guide.

Run examples

Development examples are under dev/examples and loaded with :dev.

ExampleWhat it showsOutputTypical command
benchPCAP read performance smokeednclojure -M:dev -m examples.bench
dns-summaryDNS summary rowsedn/jsonlclojure -M:dev:dns-ext -m examples.dns-summary trace.pcap
pcap-filterFilter + write + metadataedn/jsonlclojure -M:dev -m examples.pcap-filter in.pcap out.pcap 'udp and port 53' 60 jsonl --async
pcap-statsPacket and endpoint statsedn/jsonlclojure -M:dev -m examples.pcap-stats in.pcap 'udp and port 53' 10 jsonl
flow-topnTop flows (unidir/bidir)edn/jsonlclojure -M:dev -m examples.flow-topn in.pcap 'udp and port 53' 10 --async
dns-rttDNS RTT pairs/stats/qstatsedn/jsonlclojure -M:dev:dns-ext -m examples.dns-rtt in.pcap 'udp and port 53' 50 stats --async
dns-topnDNS top-N by groupedn/jsonl/csvclojure -M:dev:dns-ext -m examples.dns-topn in.pcap _ 20 rcode edn
dns-qpsDNS bucketed QPS/bytesedn/jsonl/csvclojure -M:dev:dns-ext -m examples.dns-qps in.pcap _ 1000 rcode edn
tls-sni-scanTLS ClientHello SNI top-Nedn/jsonlclojure -M:dev -m examples.tls-sni-scan in.pcap 'tcp and port 443' 10 jsonl

Notes:

  • For repository-local runs, DNS examples require -M:dev:dns-ext.
  • When used as a library dependency, DNS extension namespaces are included in the published artifact.
  • Optional positional arguments can be skipped with _.
  • format defaults to edn if omitted.
  • Bundled sample PCAPs: test/resources/dns-sample.pcap, test/resources/dns-synth-small.pcap, test/resources/tls-sni-sample.pcap.

dns-ext alias quick check

clojure -M:dev:dns-ext -e "(require '[paclo.proto.dns-ext :as dns-ext]) (dns-ext/register!) (println :dns-ext-ok)"

CLI reference

dns-topn

clojure -M:dev:dns-ext -m examples.dns-topn test/resources/dns-sample.pcap
clojure -M:dev:dns-ext -m examples.dns-topn in.pcap _ 20 qname-suffix csv --punycode-to-unicode
clojure -M:dev:dns-ext -m examples.dns-topn in.pcap _ 20 alpn edn _ --alpn-join

dns-qps

clojure -M:dev:dns-ext -m examples.dns-qps test/resources/dns-sample.pcap
clojure -M:dev:dns-ext -m examples.dns-qps in.pcap _ 200 qname edn --max-buckets 100000 --warn-buckets-threshold 50000 --emit-empty-per-key
clojure -M:dev:dns-ext -m examples.dns-qps in.pcap _ 500 rrtype jsonl --async --async-mode dropping --async-buffer 256

Both DNS CLIs use bounded analysis: RTT delegates strict transaction matching to paclo.dns; QPS counts DNS packets through paclo.window. --reorder-us controls the event-time watermark. Exact statistics and finalized output have finite limits and fail before output on overflow. See DNS CLI reporting for defaults, cancellation/drop metadata and intentional migration changes.

dns-rtt

clojure -M:dev:dns-ext -m examples.dns-rtt in.pcap
clojure -M:dev:dns-ext -m examples.dns-rtt in.pcap 'udp and port 53' 50 pairs _ edn _ --server 1.1.1.1:53
clojure -M:dev:dns-ext -m examples.dns-rtt in.pcap 'udp and port 53' 20 qstats p95 jsonl --client 192.168.4.28

pcap-stats

clojure -M:dev -m examples.pcap-stats in.pcap
clojure -M:dev -m examples.pcap-stats in.pcap 'udp and port 53' 10 jsonl

flow-topn

The CLI keeps packets and bytes metrics (bytes means captured bytes), EDN and JSONL rows, and async buffer/timeout options. Exact aggregation now defaults to at most 100000 unique flows; --max-flows N sets a positive finite budget. Existing flows can update at the limit; a new flow beyond it throws. Large captures that previously accumulated without a bound must select a larger budget. Tied rows now follow canonical numeric key order.

clojure -M:dev -m examples.flow-topn in.pcap 'udp and port 53' 10 bidir bytes jsonl --max-flows 100000
clojure -M:dev -m examples.flow-topn in.pcap 'udp or tcp' 10 bidir bytes edn --async --async-mode dropping --async-timeout-ms 1000

pcap-filter

clojure -M:dev -m examples.pcap-filter in.pcap out.pcap
clojure -M:dev -m examples.pcap-filter in.pcap out-dns.pcap 'udp and port 53' 0 jsonl
clojure -M:dev -m examples.pcap-filter /tmp/large.pcap /tmp/out.pcap --async --async-mode dropping --async-buffer 16

tls-sni-scan

clojure -M:dev -m examples.tls-sni-scan in.pcap
clojure -M:dev -m examples.tls-sni-scan in.pcap 'tcp and port 443' 10 jsonl

For full argument tables and behavior details, see docs/usage.md.

Bounded event-time analysis (v1.4.0)

paclo.window and paclo.dns provide pure immutable init, step, advance and finish APIs. Supply integer microsecond event times and monotonic watermarks; configure explicit retention bounds. DNS normalization reuses full decode and optional DNS annotations, with question-aware correlation, retry counts, timeout and bounded ID-reuse quarantine. Missing questions and unmatched responses produce explicit classifications. Capture, clocks and output remain caller-owned.

(require '[paclo.window :as window])
(let [state (window/init {:width-us 1000000 :max-windows 10 :max-groups 100})
      next (window/step state {:time-us 999999 :group :dns :captured-bytes 128})]
  (:events (window/advance (:state next) 1000000)))

See executable PCAP examples and the API contract. Repository DNS RTT/QPS CLIs now use these APIs with finite report limits, strict matching and explicit completion diagnostics; see migration details.

Public API surface (v1.x)

NamespacePublic functionsNotes
paclo.corebpf, packets, packet-xf, reduce-packets, reduce-packets-report, start-capture, capture-packets, stop-capture!, capture-stats, write-pcap!, list-devicesUser-facing API
paclo.flowkey, aggregate-rf, top-nBounded flow analysis (v1.4.0 P06)
paclo.windowinit, step, advance, finishBounded tumbling windows (v1.4.0 P07-A)
paclo.dnspacket-time-us, packet->event, init, step, advance, finishPure DNS analysis (v1.4.0 P07-A)
paclo.streamfan-out, branch, statsBounded multi-consumer distribution
paclo.pcapngopen-source, source-info, write-pcapng!Bounded PCAPNG metadata/exact-time I/O (v1.4.0 P08-E2/E3)
paclo.decode-extregister!, unregister!, installed, apply!Post-decode hook API

Internal namespaces (paclo.pcap, paclo.parse, paclo.proto.*, paclo.flow.project, paclo.linktype, paclo.stream.impl, paclo.pcapng.cursor, paclo.pcapng.writer) are not part of the compatibility contract, except supported paclo.proto.dns-ext/annotate-dns and paclo.proto.tls-ext/annotate-tls-sni.

Compatibility matrix

LayerSupportedCI gate status (2026-10-17)Notes
Clojure1.12.xRequiredBaseline in deps.edn is 1.12.1
JDK17, 21RequiredRepresentative compatibility jobs cover both
OSmacOS, LinuxRequiredmacos-latest and Linux runners
CPUx86_64, arm64RequiredRunner architecture is asserted before tests
Babashka1.12.xCheckedCI validates bb --version
libpcapSystem packageCheckedLinux uses libpcap-dev; macOS uses system pcap

The representative matrix is Linux x86_64/JDK 21, macOS arm64/JDK 17, and Linux arm64/JDK 21. The architecture assertions prevent floating runner labels from silently changing the coverage represented by a job name.

The v1.4.0 release compiles bindings with --release 17 (class-file major 61), including when built on JDK 21. Two checkout-free CI consumers run the same JDK 21-built JAR on JDK 17 and 21. The published v1.3.0 JAR contains major-65 classes and requires JDK 21; JDK 17 users of v1.3.0 must build from source. See distribution evidence.

Performance baselines

Reference measurements (:xform = drop packets shorter than 60 bytes):

SamplePacketsdecode?=falsedecode?=true
test/resources/dns-sample.pcap411.1ms13.3ms
synthetic 50k (/tmp/paclo-mid-50k.pcap)50,000273.7ms879.9ms
synthetic 100k (/tmp/bench-100k.pcap)100,000398.5ms1291.7ms

Environment: macOS 14.4 / Intel i7-8700B / JDK 21.

Perf gate:

clojure -M:perf-gate

Current thresholds: warn=1000ms, fail=1200ms.

The phase-1 runner measures offline stages independently and supports loopback-live capture with libpcap drop counters:

clojure -M:perf --mode offline --profile quick --output target/perf-quick
clojure -M:perf --mode live --profile quick --device lo0 --output target/perf-live-quick

The live runner compares direct, bounded-blocking, and bounded-dropping paths and reports sender, libpcap, queue, and consumer stages separately. It writes schema-versioned EDN and JSON results. An exact-count :perf-generator alias supports end-to-end loss measurement from a separate host over a real NIC. See docs/performance.md for profiles, coordinated external-generator measurement, metrics, and platform caveats.

Security scan (NVD)

CI runs Dependency Audit with NVD_API_TOKEN.

Local reproduction:

NVD_API_TOKEN=<token> clojure -M:nvd dev/nvd-clojure.edn "$(clojure -Spath -A:dev:dns-ext)"

Documentation

Maintainer notes

For development setup, Codex instructions, and the accepted improvement plan, see CONTRIBUTING.md.

Clojars publish automation is defined in .github/workflows/publish.yml. Required repository secrets:

  • CLOJARS_USERNAME
  • CLOJARS_PASSWORD

FAQ

Q. Live capture returns Permission denied. A. On Linux/macOS, packet capture may require root privileges or specific libpcap permissions.

Q. What happens with :decode? true on parse errors? A. Packet decode failures do not throw. Each packet includes :decode-error instead.

Q. Is this optimized for large captures? A. Yes for streaming workflows, but use BPF + :xform aggressively to reduce allocations.

License

MIT

Can you improve this documentation? These fine people already did:
nanto & NantoEdit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
Raise an issueBrowse cljdoc sourceChat on Slack
× close