clj-protobuf

The protobuf runtime for code generated by protoc-gen-clojure: records in, protoc's bytes out. Editions supported through 2024.

;; deps.edn
com.github.bpalermo/clj-protobuf {:mvn/version "0.2.4"}

What it is

protoc-gen-clojure emits a defrecord per message plus X->proto / proto->X conversion fns; this library is everything those fns call — the descriptor machinery, the field codec, and byte-level encode/decode:

(require '[clj-protobuf.core :as pb]
         '[acme.greeter.greeter :as g])   ; generated

(-> (g/map->HelloRequest {:name "world" :repeat-count 2})
    g/HelloRequest->proto
    pb/encode)                            ; -> bytes, byte-identical to protoc's

(->> bytes
     (pb/decode g/HelloRequest-prototype)
     g/proto->HelloRequest)               ; -> record; absent fields are nil

Records and plain maps are interchangeable everywhere a message value goes.

Semantics worth knowing

• nil means absent, both directions. Setting nil sets nothing; reading an unset explicit-presence field returns nil. Fields with IMPLICIT presence (editions) and proto3 no-label scalars have no absence — they read back as their value, default included.
• The proto field name is the authority. Kebab-cased keys are derived from it (camelCaseField → :camel-case-field); STYLE_LEGACY files work because nothing ever converts backwards.
• Enums are keywords of the exact proto value name (:COLOR_RED), lossless in both directions; numbers, strings and EnumValueDescriptors are accepted on the way in.
• Editions are the descriptor's problem. The generated file embeds its FileDescriptorProto; protobuf-java resolves presence, DELIMITED encoding and utf8 validation when the descriptor builds. This library carries no edition-specific code, which is why new editions need a protobuf-java bump and nothing else.
• Unknown fields survive on parsed Messages but are necessarily dropped by a record round trip; pb/unknown-fields inspects them when it matters.

Performance

Generated code carries a Java-class hint per message. When the matching java_proto_library classes are on the classpath the prototypes silently switch to the generated classes — protoc's own serializer. Without them, since 0.2.0, the prototype is this library's own compiled codec: the descriptor compiled once into reader and writer tables over a slot array, with none of DynamicMessage's per-call reflection behind it. Every arm produces byte-for-byte identical output (the byte-identity and equivalence suites prove all of them against protoc's own Java backend), and -Dclj-protobuf.codec=dynamic brings DynamicMessage back if you ever need the reference implementation.

Measured with bazel run //bench:run -- quick (criterium; JDK 21, Linux x86_64; mean latency / allocated bytes per op; full Clojure-data-to-bytes pipelines). java is protoc's generated builders driven directly — and, on decode, parsing only, since protoc's classes are not a Clojure-data path; hinted is clj-protobuf with the generated classes on the classpath; interop is protoc-gen-clojure's interop=true output, the same prototypes as hinted driven through direct Java accessors instead of the codec; compiled is clj-protobuf without the generated classes; jsonista and data.json carry the same value as JSON.

This repository builds its own library with Clojure's direct linking on, so these numbers include it — 5-18% depending on shape, and the lever described below rather than a separate one, so do not count it twice. A consumer running the same code without it is a few percent slower on every clj-protobuf row:

Encode (Clojure data → bytes)

shape	java	hinted	interop	compiled	jsonista	data.json
tiny	67 ns / 96 B	152 ns / 136 B	85 ns / 136 B	236 ns / 192 B	347 ns / 608 B	823 ns / 624 B
flat	602 ns / 400 B	978 ns / 488 B	627 ns / 400 B	905 ns / 440 B	1.09 µs / 1264 B	2.93 µs / 2208 B
deep	—	756 ns / 552 B	384 ns / 368 B	862 ns / 560 B	658 ns / 1048 B	2.62 µs / 1392 B
wide-repeated	—	3.51 µs / 3088 B	3.33 µs / 3088 B	2.64 µs / 2088 B	2.54 µs / 1112 B	6.44 µs / 4216 B
repeated-messages	2.07 µs / 2344 B	4.44 µs / 2824 B	4.38 µs / 2824 B	5.36 µs / 3000 B	3.80 µs / 4040 B	16.30 µs / 10424 B
map-heavy	—	8.05 µs / 7512 B	7.55 µs / 7512 B	5.74 µs / 4736 B	3.70 µs / 3632 B	11.98 µs / 10720 B
enum-heavy	—	1.31 µs / 480 B	1.30 µs / 480 B	1.10 µs / 336 B	1.43 µs / 1184 B	3.92 µs / 2952 B

Decode (bytes → Clojure data)

shape	java	hinted	interop	compiled	jsonista	data.json
tiny	67 ns / 192 B	176 ns / 232 B	102 ns / 232 B	205 ns / 216 B	628 ns / 1136 B	659 ns / 1584 B
flat	348 ns / 432 B	771 ns / 600 B	455 ns / 600 B	539 ns / 520 B	1.98 µs / 2168 B	2.98 µs / 5776 B
deep	—	860 ns / 1024 B	355 ns / 1024 B	671 ns / 760 B	1.28 µs / 2128 B	1.24 µs / 3840 B
wide-repeated	—	2.80 µs / 4352 B	2.85 µs / 4352 B	2.90 µs / 4208 B	3.43 µs / 4144 B	3.05 µs / 11496 B
repeated-messages	1.77 µs / 3312 B	5.73 µs / 5344 B	2.55 µs / 5336 B	4.72 µs / 4704 B	10.81 µs / 10920 B	9.95 µs / 25096 B
map-heavy	—	12.21 µs / 13048 B	11.84 µs / 13032 B	10.68 µs / 10032 B	8.55 µs / 5776 B	13.97 µs / 22248 B
enum-heavy	—	2.17 µs / 1056 B	1.29 µs / 1056 B	1.95 µs / 856 B	2.22 µs / 2880 B	2.57 µs / 7200 B

quick trades accuracy for time, and the sub-100-ns cells move between runs of it — the java column on tiny decode has come out anywhere from 65 to 99 ns. Under the full bazel run //bench:run, Tiny/parseFrom measured 76.7 ns and 66.8 ns in one JVM — the same call, twice — with interop's whole decode pipeline at 71.4 ns against it and the hinted arm at 119.9 ns. Read the small shapes as "interop reaches protoc's parse floor", not as a ranking between the two.

One more lever, measured on a real gRPC service rather than here: this jar ships as source, so Clojure compiles these namespaces when they load, and -Dclojure.compiler.direct-linking=true on the JVM turns every call between them into a static call. On a 1-CPU gRPC server echoing 1 KB messages that was worth 5–17% of CPU per request depending on shape and rate, and removed a load-shedding cliff at the top of the ramp. The property is process-wide and changes late binding for everything loaded from source — with-redefs on a linked call site stops taking effect — so turn it on deliberately, in production images rather than at the REPL.

A Bazel consumer has a second route: from rules_clj 0.2.5 a source-only dependency's namespaces can be compiled in the consumer's own build, which is what lets a direct-linked target call into a library published as source. That is rules_clj's mechanism rather than this library's, and its docs are the place to read it; the jar published here stays source either way, so that generated records and call sites specialize against the Clojure and protobuf-java versions the consumer actually builds with.

Read it honestly. Against protoc's own generated code the hinted arm is roughly 2× on tiny messages and closer on wider ones; the compiled arm is within 1.6× of the hinted arm at worst and beats it on most decode shapes, because a slot read is cheaper than a typed-accessor call. interop=true beats the hinted arm everywhere except the two collection-heavy decode rows, where they are a wash — building the Clojure collection is the row, not the accessor — and it allocates exactly what the hinted arm does, since both produce the same Clojure values from the same generated classes. Against JSON, protobuf wins both directions on the small and nested shapes and wins decode on lists of messages, while jackson wins encoding every collection-heavy shape and also wins map-heavy decode — building a 50-entry Clojure map is most of that row, and protobuf pays for entry messages on top. Before 0.2.0 the arm without generated classes was DynamicMessage, two to three times slower than the compiled one on decode. Wire compactness and schema are protobuf's argument regardless, and the shapes are archetypes precisely because no single number describes "protobuf vs JSON".

The interop column needs protoc's Java classes at load time — that is the interop=true contract — and it is emitted code, not a runtime switch: it comes from protoc-gen-clojure 0.6.0 or later, which emits direct accessor calls in both directions (writes were typed earlier). //test:interop_test holds it to the same bytes and the same values as the codec path, and //bench:smoke_test holds every arm in these tables to byte-identical output.

Building

Bazel (with rules_clj) is the build and test harness: bazel test //.... Plain clj works too: clojure -X:test. The Clojars artifact is bazel build //src:clojars — jar and pom, from the same deps.edn and version.edn everything else here reads. bazel run //src:clojars.publish -- --dry-run prints every upload it would make.

License

Apache-2.0