clj-protobuf

The protobuf runtime for code generated by protoc-gen-clojure: records in, protoc's bytes out. Editions supported through 2024.

;; deps.edn
com.github.bpalermo/clj-protobuf {:mvn/version "0.2.0"}

What it is

protoc-gen-clojure emits a defrecord per message plus X->proto / proto->X conversion fns; this library is everything those fns call — the descriptor machinery, the field codec, and byte-level encode/decode:

(require '[clj-protobuf.core :as pb]
         '[acme.greeter.greeter :as g])   ; generated

(-> (g/map->HelloRequest {:name "world" :repeat-count 2})
    g/HelloRequest->proto
    pb/encode)                            ; -> bytes, byte-identical to protoc's

(->> bytes
     (pb/decode g/HelloRequest-prototype)
     g/proto->HelloRequest)               ; -> record; absent fields are nil

Records and plain maps are interchangeable everywhere a message value goes.

Semantics worth knowing

• nil means absent, both directions. Setting nil sets nothing; reading an unset explicit-presence field returns nil. Fields with IMPLICIT presence (editions) and proto3 no-label scalars have no absence — they read back as their value, default included.
• The proto field name is the authority. Kebab-cased keys are derived from it (camelCaseField → :camel-case-field); STYLE_LEGACY files work because nothing ever converts backwards.
• Enums are keywords of the exact proto value name (:COLOR_RED), lossless in both directions; numbers, strings and EnumValueDescriptors are accepted on the way in.
• Editions are the descriptor's problem. The generated file embeds its FileDescriptorProto; protobuf-java resolves presence, DELIMITED encoding and utf8 validation when the descriptor builds. This library carries no edition-specific code, which is why new editions need a protobuf-java bump and nothing else.
• Unknown fields survive on parsed Messages but are necessarily dropped by a record round trip; pb/unknown-fields inspects them when it matters.

Performance

Generated code carries a Java-class hint per message. When the matching java_proto_library classes are on the classpath the prototypes silently switch to the generated classes — protoc's own serializer. Without them, since 0.2.0, the prototype is this library's own compiled codec: the descriptor compiled once into reader and writer tables over a slot array, with none of DynamicMessage's per-call reflection behind it. Every arm produces byte-for-byte identical output (the byte-identity and equivalence suites prove all of them against protoc's own Java backend), and -Dclj-protobuf.codec=dynamic brings DynamicMessage back if you ever need the reference implementation.

Measured with bazel run //bench:run -- quick (criterium; JDK 21, Linux x86_64; mean latency / allocated bytes per op; full Clojure-data-to-bytes pipelines). java is protoc's generated builders driven directly; hinted is clj-protobuf with the generated classes on the classpath; compiled is clj-protobuf without them; jsonista and data.json carry the same value as JSON:

Encode (Clojure data → bytes)

shape	java	hinted	compiled	jsonista	data.json
tiny	86 ns / 96 B	168 ns / 136 B	311 ns / 224 B	390 ns / 608 B	1.03 µs / 624 B
flat	682 ns / 312 B	1.18 µs / 488 B	1.36 µs / 504 B	1.57 µs / 1248 B	4.01 µs / 2208 B
deep	—	907 ns / 552 B	1.27 µs / 656 B	1.14 µs / 1080 B	2.83 µs / 1392 B
wide-repeated	—	4.26 µs / 3088 B	3.77 µs / 2112 B	2.63 µs / 1096 B	8.65 µs / 4216 B
repeated-messages	2.25 µs / 2296 B	6.42 µs / 2824 B	7.13 µs / 3664 B	4.16 µs / 4024 B	24.40 µs / 10424 B
map-heavy	—	10.61 µs / 7512 B	8.48 µs / 4760 B	4.46 µs / 3600 B	16.60 µs / 10720 B
enum-heavy	—	1.44 µs / 456 B	1.36 µs / 392 B	1.61 µs / 1168 B	5.44 µs / 2952 B

Decode (bytes → Clojure data)

shape	java	hinted	compiled	jsonista	data.json
tiny	97 ns / 192 B	217 ns / 232 B	242 ns / 216 B	769 ns / 1136 B	761 ns / 1584 B
flat	407 ns / 432 B	910 ns / 600 B	638 ns / 520 B	2.10 µs / 2168 B	4.86 µs / 5776 B
deep	—	918 ns / 1024 B	744 ns / 760 B	1.32 µs / 2088 B	1.19 µs / 3840 B
wide-repeated	—	3.87 µs / 4352 B	3.88 µs / 4208 B	3.33 µs / 4104 B	5.73 µs / 11496 B
repeated-messages	2.04 µs / 3312 B	6.10 µs / 5344 B	6.18 µs / 4704 B	11.60 µs / 10080 B	10.45 µs / 25096 B
map-heavy	—	13.73 µs / 13048 B	11.52 µs / 10032 B	10.11 µs / 5736 B	15.18 µs / 22248 B
enum-heavy	—	1.99 µs / 1056 B	1.87 µs / 856 B	2.22 µs / 2840 B	2.69 µs / 7200 B

Read it honestly: the hinted arm sits ~2× off protoc's own generated code on small messages and the compiled arm within ~1.3× of the hinted one — and on decode the compiled arm is the faster of the two on most shapes, a slot read beating a typed-accessor call. Both beat JSON both ways on small and nested messages, and lists of messages decode faster than JSON too; jackson still wins encoding the collection-heavy shapes, where protobuf-java's own map and list building dominates. Before 0.2.0 the arm without generated classes was DynamicMessage, two to three times slower than the compiled one on decode and up to 2.5× on encode. Wire compactness and schema are protobuf's argument regardless. The shapes are archetypes precisely because no single number describes 'protobuf vs JSON'.

Building

Bazel (with rules_clj) is the build and test harness: bazel test //.... Plain clj works too: clojure -X:test. The Clojars artifact is bazel build //src:clojars — jar and pom, from the same deps.edn and version.edn everything else here reads. bazel run //src:clojars.publish -- --dry-run prints every upload it would make.

License

Apache-2.0