clj-protobuf — design

The protobuf runtime for code generated by protoc-gen-clojure. The generator emits a defrecord per message and thin conversion fns; everything those fns call lives here. That split is deliberate and directional: the plugin depends on nothing but protobuf-java and owns emission; this library owns behavior, including the behavioral tests of emitted code — which is what keeps the two dependency graphs acyclic.

The contract

Generated code calls exactly six symbols from this library, with hardcoded aliases:

call	provides
rt/file-descriptor b64 [deps…]	rebuild the embedded FileDescriptorProto, linked against dependency FileDescriptors
rt/known-file "google/protobuf/…"	well-known-type descriptors bundled in protobuf-java
rt/message fd "Outer.Inner" hint?	a prototype per message (see pools, below)
rt/field prototype "proto_name"	a precomputed FieldHandle per field
codec/set-field! b handle v opts	one field, record/map → builder
codec/get-field msg handle opts	one field, message → Clojure value

Everything else — clj-protobuf.core's encode/decode, opts, errors — is public API on top of the contract, not part of it. The contract is versioned by the emitter's own comments: the 3-arg rt/message class hint needs 0.1.3, dotted nested lookup needs 0.1.5; the first published artifact is 0.1.5 so both floors name a version that exists.

clj-grpc.codec and clj-grpc.runtime ship here as deprecated def-aliases: plugin ≤ 0.3.x emitted those namespace names, and message-only files must keep working without any gRPC artifact on the classpath. The clj-grpc artifact must never define them, or the two jars collide. Removed at 1.0.

Editions: the descriptor's problem, on purpose

The generated file embeds its FileDescriptorProto verbatim; protobuf-java resolves every edition feature — presence, DELIMITED message encoding, utf8 validation, packedness — when FileDescriptor/buildFrom runs. This library performs no feature resolution and carries no edition-specific code, which is why edition 2024 support is a protobuf-java version (4.35.x) rather than a feature of this codebase, and why the next edition should cost a dependency bump and nothing else. The fixture suite pins this across proto2, proto3, editions 2023/2024, DELIMITED, IMPLICIT presence and STYLE_LEGACY.

Descriptor pools: never mix them

rt/message's third argument is a Java class hint. When the class exists and its descriptor's full name matches, the prototype is the generated class's default instance — measured ~2.4× faster to encode with ~3× less allocation on small messages than DynamicMessage, with byte-identical output (the byte-identity suite proves both arms against protoc's own Java backend). Any hint failure is silent: being wrong costs the optimisation, never the bytes.

The consequence worth a rule: the hinted prototype lives in the generated classes' descriptor pool, while the file-descriptor var builds a separate pool from the embedded bytes — and protobuf-java forbids using one pool's FieldDescriptor against the other's messages. Generated code is immune by construction (every handle chains off the prototype), but anything else that manufactures prototypes for the same types — clj-grpc's marshallers, say — must resolve them the same way, hint first, same fallback. The wire is where pools meet; field access is where they must not.

FieldHandle: pay at def-time, not per call

rt/field returns a record precomputing everything the codec's hot path needs: a kind keyword to case on, repeated/map flags, presence, the enum type, map key/value handles, and — for message fields — a nested prototype obtained through newBuilderForField, so it has the right concrete class for this prototype's lineage in both pools, plus delayed child handles (delayed because descriptors can be cyclic). Generated code stores handles in vars, so the descriptor API is walked once per field per namespace load, and the codec never touches it again.

Hinted-arm handles for singular non-enum fields also carry typed-accessor invokers: LambdaMetafactory-generated functions over the generated class's setX/getX/hasX, built (and verified via findVirtual) at handle time, measured at direct-interop speed. The accessor name is derived by protoc's UnderscoresToCamelCase rule, and every failure — underivable name, protoc's conflict-mangled accessors, LambdaMetafactory being unavailable as it is under native-image — silently yields no invoker and the reflection path serves, same philosophy as the class hint: wrong is never incorrect, only unoptimized. Repeated and map fields instead batch into a single setField/getField (one accessor lookup per field); enums stay reflective. The whole library compiles reflection-free; a CI gate recompiles every namespace under *warn-on-reflection* and fails on a single warning, because one reflective call site on this path silently costs an order of magnitude.

Semantics

• nil means absent, both directions. set-field! of nil sets nothing; get-field of an unset explicit-presence field returns nil. IMPLICIT presence (and proto3 no-label scalars) has no absence: reads return the value, default included. Empty repeated/map fields read as nil.
• The proto field name is the authority. Kebab keys are derived by the emitter's exact algorithm (ported byte-for-byte in impl/naming.clj, and frozen — changing it is a wire break with every generated file); the reverse mapping deliberately does not exist, which is what makes STYLE_LEGACY files work.
• Records and plain maps are interchangeable everywhere a message value goes; nested messages read back as plain kebab maps (this runtime cannot know the generated record classes).
• Enums are keywords of the exact proto value name, lossless; numbers, strings and EnumValueDescriptors are accepted on the way in, with findValueByNumberCreatingIfUnknown covering open-enum unknowns.
• Unknown fields survive on parsed Messages, are necessarily dropped by a record round trip, and are inspectable via core/unknown-fields.
• Errors are ex-info with {:clj-protobuf/error <category>} — :parse, :type-mismatch (naming the field), :no-such-field, :no-such-type, :descriptor — so callers dispatch on data, not message strings.

Build and release

Bazel + rules_clj is the build and test harness (this repo and clj-grpc are the ruleset's first library-shaped consumers, deliberately); deps.edn is the single dependency source, consumed by Bazel through the committed deps.lock.json and by Clojars consumers through the pom that build.clj generates from the same file. The jar is source-only: Clojure libraries compile in the consumer's process, so records specialize against the consumer's own dependency versions.

Gates, all riding bazel test //...: the contract and byte-identity suites, the error suite, the reflection gate, buildifier formatting, and a version-consistency test keeping the README install snippet equal to version.edn — the one version copy no other machine checks. CI adds a plain-clj leg (clojure -X:test, clojure -T:build jar) proving the non-Bazel consumer path; the release workflow refuses tags that are not on main or disagree with version.edn before anything can reach the Clojars token, which lives in a release environment that only v* tags may enter.

The benchmark (bazel run //bench:run) is manual, with a smoke test keeping its arms compiling and byte-agreeing in CI; the corpus is six archetype shapes because no single number describes "protobuf vs JSON", and the README table reports where JSON wins (collection-heavy shapes) alongside where it loses.

Non-goals

• Schema-free maps-to-bytes. Everything goes through a descriptor; the generated code is the API.
• A gRPC surface. Message-only consumers never drag grpc-java onto their classpath; that boundary is the reason clj-grpc is a separate artifact.
• Reimplementing the wire format. protobuf-java is the wire; this library is the shape of its use from Clojure.
• Custom outer-class naming rules. The class hint covers what protoc generates for java_multiple_files and edition-2024 top-level classes; the pre-2024 OuterClass collision rules are not reproduced (a missing hint is only a missed optimisation).