clj-protobuf — design

The protobuf runtime for code generated by protoc-gen-clojure. The generator emits a defrecord per message and thin conversion fns; everything those fns call lives here. That split is deliberate and directional: the plugin depends on nothing but protobuf-java and owns emission; this library owns behavior, including the behavioral tests of emitted code — which is what keeps the two dependency graphs acyclic.

The contract

Generated code calls exactly six symbols from this library, with hardcoded aliases:

call	provides
rt/file-descriptor b64 [deps…]	rebuild the embedded FileDescriptorProto, linked against dependency FileDescriptors
rt/known-file "google/protobuf/…"	well-known-type descriptors bundled in protobuf-java
rt/message fd "Outer.Inner" hint?	a prototype per message (see pools, below)
rt/field prototype "proto_name"	a precomputed FieldHandle per field
codec/set-field! b handle v opts	one field, record/map → builder
codec/get-field msg handle opts	one field, message → Clojure value

Everything else — clj-protobuf.core's encode/decode, opts, errors — is public API on top of the contract, not part of it. The contract is versioned by the emitter's own comments: the 3-arg rt/message class hint needs 0.1.3, dotted nested lookup needs 0.1.5; the first published artifact is 0.1.5 so both floors name a version that exists.

clj-grpc.codec and clj-grpc.runtime ship here as deprecated def-aliases: plugin ≤ 0.3.x emitted those namespace names, and message-only files must keep working without any gRPC artifact on the classpath. The clj-grpc artifact must never define them, or the two jars collide. Removed at 1.0.

Editions: the descriptor's problem, on purpose

The generated file embeds its FileDescriptorProto verbatim; protobuf-java resolves every edition feature — presence, DELIMITED message encoding, utf8 validation, packedness — when FileDescriptor/buildFrom runs. This library performs no feature resolution and carries no edition-specific code, which is why edition 2024 support is a protobuf-java version (4.35.x) rather than a feature of this codebase, and why the next edition should cost a dependency bump and nothing else. The fixture suite pins this across proto2, proto3, editions 2023/2024, DELIMITED, IMPLICIT presence and STYLE_LEGACY.

Descriptor pools: never mix them

rt/message's third argument is a Java class hint. When the class exists and its descriptor's full name matches, the prototype is the generated class's default instance — protoc's own serializer, the fastest arm, with byte-identical output (the byte-identity suite proves every arm against protoc's own Java backend). Any hint failure is silent: being wrong costs the optimisation, never the bytes. Without a usable hint the prototype is the compiled codec's (next section); DynamicMessage serves only extendable types and -Dclj-protobuf.codec=dynamic.

The consequence worth a rule: the hinted prototype lives in the generated classes' descriptor pool, while the file-descriptor var builds a separate pool from the embedded bytes — and protobuf-java forbids using one pool's FieldDescriptor against the other's messages. Generated code is immune by construction (every handle chains off the prototype), but anything else that manufactures prototypes for the same types — clj-grpc's marshallers, say — must resolve them the same way, hint first, same fallback. Since 0.2.1 that is one call: rt/prototype takes a bare Descriptor (or any Message of the type) and returns the arm the generated namespace got, deriving the class hint by the emitter's own rule (rt/java-class-hint, a port of it, which reads nest_in_file_class both as an unknown field and as a known extension, because in the consumer's JVM a loaded generated class makes it the latter). A DynamicMessage passed in is re-resolved, so a library that built one before 0.2.0 wraps that call and changes nothing else. The wire is where pools meet; field access is where they must not.

The compiled codec

Before 0.2.0 the arm without generated classes was DynamicMessage, and a profile of a gRPC service on it said where the time went: a quarter to a third of CPU per request was DynamicMessage's FieldSet and SmallSortedMap, protobuf-java resolving edition features on every FieldDescriptor.getType, and the reflective accessor lookups — the representation, not the wire parsing, which was 14–21%. None of that is reachable from the codec fns: generated ->proto code obtains its builder from the prototype rt/message returned, so a runtime-only fix had to replace the prototype.

It does. rt/message now returns a Message implemented here (impl/message.clj): a compiled type, an Object[] of slots — one per field, in descriptor order, nil for absent — and an UnknownFieldSet. Its builder is the same, mutable; its parser owns fresh slots. The compiled type (impl/compile.clj) is the descriptor walked once, the first time the type is used: a writer per field over CodedOutputStream, in field-number order; a reader table keyed by tag, dense when the field numbers allow and binary-searched otherwise, in which a repeated scalar registers both its packed and its expanded tag because a parser accepts either; oneof membership, so a member read off the wire clears its siblings; the required slots. Every edition feature that changes bytes — packed, DELIMITED, IMPLICIT presence, utf8 validation, closed enums — is decided here from the resolved descriptor and never consulted again. The primitives themselves (impl/wire.clj) stay protobuf-java's: varints, zigzag, fixed widths, UTF-8 and the length arithmetic are CodedInputStream and CodedOutputStream calls chosen once per field instead of once per value. Nested types compile lazily behind an IDeref, which is what makes cyclic descriptors terminate.

Three rules keep the arms interchangeable. Fields without presence are normalized so nil also means the default — readers and setters store nil for the default value, reads substitute it back — which is what makes re-encoding bytes that carried an explicit default drop it, as every protobuf implementation does. Collections in slots are never mutated in place once a message may share them: a builder that has built, or came from toBuilder, copies before touching one. And the reflective API returns exactly what protobuf-java's does — EnumValueDescriptor for enums, entry messages for maps, nested default instances for unset message fields — with equals, hashCode and toString following AbstractMessage's algorithm, so a compiled message equals and hashes like a DynamicMessage of the same value, TextFormat prints and parses it, and grpc-java's marshaller — which trusts getSerializedSize() and then writeTo(OutputStream) — is served byte for byte. The codec bypasses all of that: on a compiled prototype a handle carries its slot index, set-field! coerces and stores the slot, get-field reads it.

The kill switch is a JVM system property, clj-protobuf.codec=dynamic, read once at load — rt/message runs when a generated namespace loads, under AOT or inside a native image, where binding a Var first is impractical — and it makes a soak A/B a one-line environment change. rt/dynamic-message and rt/compiled-message hand out either arm explicitly, which is how the equivalence suite drives all three through the wire corpus and values generated from the descriptors themselves.

FieldHandle: pay at def-time, not per call

rt/field returns a record precomputing everything the codec's hot path needs: a kind keyword to case on, repeated/map flags, presence, the enum type, map key/value handles, on the compiled arm the slot index and the default in slot representation, and — for message fields — a nested prototype of this prototype's own lineage (through newBuilderForField for generated and dynamic parents, straight from the compiler for compiled ones, so that a map's entry type and its children are compiled too), plus delayed child handles (delayed because descriptors can be cyclic). Generated code stores handles in vars, so the descriptor API is walked once per field per namespace load, and the codec never touches it again.

Hinted-arm handles also carry typed-accessor invokers: LambdaMetafactory-generated functions over the generated class's accessors, built (and verified via findVirtual) at handle time, measured at direct-interop speed. Singular scalars and messages use setX/getX/hasX. Repeated fields use clearX + addAllX and getXList; maps use clearX + putAllX and getXMap — clear first, because the bulk accessors append and merge where setField replaces, and set-field! replaces. Open enums (every proto3 and editions enum) use setXValue(int)/getXValue(), so an enum field costs what an int field costs; closed proto2 enums have no number accessors and stay reflective, as do repeated enums and enum-valued maps, whose bulk accessors take generated Java enum classes. The accessor name is derived by protoc's UnderscoresToCamelCase rule, and every failure — underivable name, protoc's conflict-mangled accessors, LambdaMetafactory being unavailable as it is under native-image — silently yields no invoker and the reflection path serves, same philosophy as the class hint: wrong is never incorrect, only unoptimized.

Two more things are paid at def-time. Enum lookups in every direction are tables on the handle (keyword → value, number → value, value → keyword), because EnumDescriptor.findValueByName is a string concatenation plus a pool lookup per call. And a nested message's children are an array of handles walked by index; reading one back gathers the present fields into a single key/value array and builds the map in one step, which halved decode time on lists of small messages against the reduce/assoc shape.

One ordering rule falls out of the map accessors: protobuf-java serializes map entries in insertion order, and both arms insert in the Clojure map's iteration order — the reflection arm through its entry list, the invoker arm through a LinkedHashMap. That is what keeps the two arms byte-identical on maps; a HashMap there changes the bytes.

The whole library compiles reflection-free; a CI gate recompiles every namespace under *warn-on-reflection* and fails on a single warning, because one reflective call site on this path silently costs an order of magnitude.

Semantics

• nil means absent, both directions. set-field! of nil sets nothing; get-field of an unset explicit-presence field returns nil. IMPLICIT presence (and proto3 no-label scalars) has no absence: reads return the value, default included. Empty repeated/map fields read as nil.
• The proto field name is the authority. Kebab keys are derived by the emitter's exact algorithm (ported byte-for-byte in impl/naming.clj, and frozen — changing it is a wire break with every generated file); the reverse mapping deliberately does not exist, which is what makes STYLE_LEGACY files work.
• Records and plain maps are interchangeable everywhere a message value goes; nested messages read back as plain kebab maps (this runtime cannot know the generated record classes).
• Enums are keywords of the exact proto value name, lossless; numbers, strings and EnumValueDescriptors are accepted on the way in, with findValueByNumberCreatingIfUnknown covering open-enum unknowns.
• Unknown fields survive on parsed Messages, are necessarily dropped by a record round trip, and are inspectable via core/unknown-fields.
• Errors are ex-info with {:clj-protobuf/error <category>} — :parse, :type-mismatch (naming the field), :no-such-field, :no-such-type, :descriptor — so callers dispatch on data, not message strings.

Build and release

Bazel + rules_clj is the build and test harness (this repo and clj-grpc are the ruleset's first library-shaped consumers, deliberately); deps.edn is the single dependency source, consumed by Bazel through the committed deps.lock.json and by Clojars consumers through the pom that build.clj generates from the same file. The jar is source-only: Clojure libraries compile in the consumer's process, so records specialize against the consumer's own dependency versions.

Gates, all riding bazel test //...: the contract and byte-identity suites, the equivalence suite (every arm of the runtime — hinted, DynamicMessage, compiled — must produce protoc's bytes and read the same values, over a wire corpus covering every scalar wire type, packed and unpacked repeateds, closed and open enums, groups, explicit defaults, required fields, and the editions features that change bytes, plus values generated from the descriptors themselves), the compiled codec's own suites (the wire primitives, the compiler's tables and loops, and the message layer's parity with DynamicMessage, all against protobuf-java on the same bytes), the error suite, the reflection gate (over the library, and separately over the interop=true fixtures, whose whole premise is direct typed calls), a fixture drift test, buildifier formatting, and a version-consistency test keeping the README install snippet equal to version.edn — the one version copy no other machine checks.

The fixtures under test/fixtures are vendored emitter output and stay vendored: the plain-clj leg reads them from disk, a pull request shows an emitter change as a diff, and old emissions are backward-compatibility coverage. What changed is how they are refreshed. MODULE.bazel pins protoc-gen-clojure from the Bazel Central Registry as a dev dependency, //test/proto generates every variant from it (standard, the unresolvable bench_nohint hints, and interop=true), //test:update_fixtures_tests fails when a vendored file differs from that output, and bazel run //test:update_fixtures rewrites them. Bumping the pin is the one step of a fixture refresh; the diff it produces is the review. CI adds a plain-clj leg (clojure -X:test) proving the non-Bazel consumer path, and builds //src:clojars to prove the publishable jar and pom still assemble; the release workflow refuses tags that are not on main or disagree with version.edn before anything can reach the Clojars token, which lives in a release environment that only v* tags may enter.

The benchmark (bazel run //bench:run) is manual, with a smoke test keeping its arms compiling and byte-agreeing in CI; the corpus is six archetype shapes because no single number describes "protobuf vs JSON", and the README table reports where JSON wins (collection-heavy shapes) alongside where it loses.

Non-goals

• Schema-free maps-to-bytes. Everything goes through a descriptor; the generated code is the API.
• A gRPC surface. Message-only consumers never drag grpc-java onto their classpath; that boundary is the reason clj-grpc is a separate artifact.
• Reimplementing the wire format. protobuf-java is the wire; this library is the shape of its use from Clojure.
• Custom outer-class naming rules. The class hint covers what protoc generates for java_multiple_files and edition-2024 top-level classes; the pre-2024 OuterClass collision rules are not reproduced (a missing hint is only a missed optimisation).