Liking cljdoc? Tell your friends :D

clj-protobuf.codec

The codec half of the generated-code contract: set-field! and get-field.

Generated ->proto fns call set-field! once per field against a fresh builder; proto-> fns call get-field once per field feeding the record's positional constructor. Both take an opts map that is almost always nil.

Semantics the whole design hangs on:

  • nil means absent, in both directions. A record has every key; protobuf has presence. set-field! of nil sets nothing; get-field of an unset explicit-presence field returns nil. Fields with IMPLICIT presence (editions) and proto3 no-label scalars have no absence to report, so get-field returns the value — default included — and never nil.
  • Nested message values arrive as records or plain maps; generated code never calls the nested ->proto, so recursion happens here, through the handle's child handles, which carry the concrete-class-correct nested prototypes.
  • The proto field name is the authority. Kebab keys are derived; the reverse mapping does not exist (STYLE_LEGACY).

opts (all optional): :naming :kebab (default) | :proto — keys used on the generic map path :enums :keyword (default; exact proto value name, e.g. :GREETING_HELLO) | :number | :string — how get-field represents enums; set-field! accepts keyword, string, number or EnumValueDescriptor regardless :bytes :byte-array (default) | :byte-string

The codec half of the generated-code contract: `set-field!` and `get-field`.

Generated `->proto` fns call `set-field!` once per field against a fresh
builder; `proto->` fns call `get-field` once per field feeding the record's
positional constructor. Both take an opts map that is almost always nil.

Semantics the whole design hangs on:
- nil means absent, in both directions. A record has every key; protobuf has
  presence. `set-field!` of nil sets nothing; `get-field` of an unset
  explicit-presence field returns nil. Fields with IMPLICIT presence
  (editions) and proto3 no-label scalars have no absence to report, so
  `get-field` returns the value — default included — and never nil.
- Nested message values arrive as records or plain maps; generated code never
  calls the nested `->proto`, so recursion happens here, through the handle's
  child handles, which carry the concrete-class-correct nested prototypes.
- The proto field name is the authority. Kebab keys are derived; the reverse
  mapping does not exist (STYLE_LEGACY).

opts (all optional):
  :naming  :kebab (default) | :proto  — keys used on the generic map path
  :enums   :keyword (default; exact proto value name, e.g. :GREETING_HELLO)
           | :number | :string        — how get-field represents enums;
           set-field! accepts keyword, string, number or EnumValueDescriptor
           regardless
  :bytes   :byte-array (default) | :byte-string
raw docstring

clj-protobuf.core

Serialization, on top of what the generated code produces.

Generated X->proto / proto->X fns are Message-in/Message-out — they never touch bytes. This namespace is the other half: Message to bytes and back.

Typical round trip:

(-> rec HelloRequest->proto pb/encode)                     ; -> bytes
(->> bytes (pb/decode HelloRequest-prototype) proto->HelloRequest)

decode takes the generated X-prototype var — the same value everything else in the contract keys off — so there is exactly one handle per message type in user code.

Serialization, on top of what the generated code produces.

Generated `X->proto` / `proto->X` fns are Message-in/Message-out — they never
touch bytes. This namespace is the other half: Message to bytes and back.

Typical round trip:

    (-> rec HelloRequest->proto pb/encode)                     ; -> bytes
    (->> bytes (pb/decode HelloRequest-prototype) proto->HelloRequest)

`decode` takes the generated `X-prototype` var — the same value everything
else in the contract keys off — so there is exactly one handle per message
type in user code.
raw docstring

clj-protobuf.impl.compile

The codec compiler: a Descriptor walked once into everything the hot path needs, so that parsing and writing never touch the descriptor API, never resolve a FieldDescriptor, and never ask protobuf-java what an edition feature means — those questions are answered here, per type, the first time the type is used.

A message value is an Object[] of slots, one per field in descriptor order (FieldDescriptor.getIndex), plus an UnknownFieldSet. nil is absent. Fields without presence (proto3 scalars, editions IMPLICIT) are normalized so nil also means the default: readers and setters store nil for the default value, and reads substitute the default back. That keeps the writer's rule one line — write what is non-nil and non-empty — and makes re-encoding bytes that carried an explicit default drop it, as every protobuf implementation does.

Compiled per type:

  • fields: a CompiledField per slot, with the writer for that field.
  • writers: the same fields in field-number order, the order every protobuf serializer emits, followed by unknown fields.
  • a reader table keyed by TAG, not field number, dense (an array indexed by tag) when the tags allow and binary-searched otherwise. A repeated scalar registers both its packed and its expanded tag, because a parser accepts either encoding whatever the descriptor says it writes.
  • oneof membership, so reading a member off the wire clears its siblings, and the required slots, for initialization checks.

Nested types compile lazily through the parser-fn the message layer supplies, behind an IDeref, because descriptors are cyclic. One compiler owns one cache, keyed by Descriptor identity.

The codec compiler: a Descriptor walked once into everything the hot
path needs, so that parsing and writing never touch the descriptor API,
never resolve a FieldDescriptor, and never ask protobuf-java what an
edition feature means — those questions are answered here, per type, the
first time the type is used.

A message value is an Object[] of slots, one per field in descriptor
order (FieldDescriptor.getIndex), plus an UnknownFieldSet. nil is absent.
Fields without presence (proto3 scalars, editions IMPLICIT) are normalized
so nil also means the default: readers and setters store nil for the
default value, and reads substitute the default back. That keeps the
writer's rule one line — write what is non-nil and non-empty — and makes
re-encoding bytes that carried an explicit default drop it, as every
protobuf implementation does.

Compiled per type:
- `fields`: a CompiledField per slot, with the writer for that field.
- `writers`: the same fields in field-number order, the order every
  protobuf serializer emits, followed by unknown fields.
- a reader table keyed by TAG, not field number, dense (an array indexed
  by tag) when the tags allow and binary-searched otherwise. A repeated
  scalar registers both its packed and its expanded tag, because a parser
  accepts either encoding whatever the descriptor says it writes.
- oneof membership, so reading a member off the wire clears its siblings,
  and the required slots, for initialization checks.

Nested types compile lazily through the parser-fn the message layer
supplies, behind an IDeref, because descriptors are cyclic. One compiler
owns one cache, keyed by Descriptor identity.
raw docstring

clj-protobuf.impl.invoke

Typed-accessor invokers, built once per field with LambdaMetafactory.

protobuf-java's reflection API pays a FieldAccessorTable lookup on every setField/getField. When the prototype is a generated class, the typed accessors (setFooBar, getFooBar, hasFooBar) are right there — and a metafactory-generated BiFunction/Function calling one runs at direct-interop speed (measured ~3.5 ns/op, vs ~µs through reflection paths), including the primitive boxing bridge the instantiated method type declares.

Everything here is best-effort by construction, in the same spirit as the Java-class hint: derive protoc's accessor name, let findVirtual verify it exists with the expected signature, and return nil on ANY failure — including LambdaMetafactory itself being unavailable, which is what happens under native-image, where the codec silently keeps its reflection path. A wrong derivation is never wrong bytes, only a missed optimisation.

Typed-accessor invokers, built once per field with LambdaMetafactory.

protobuf-java's reflection API pays a FieldAccessorTable lookup on every
setField/getField. When the prototype is a generated class, the typed
accessors (setFooBar, getFooBar, hasFooBar) are right there — and a
metafactory-generated BiFunction/Function calling one runs at direct-interop
speed (measured ~3.5 ns/op, vs ~µs through reflection paths), including the
primitive boxing bridge the instantiated method type declares.

Everything here is best-effort by construction, in the same spirit as the
Java-class hint: derive protoc's accessor name, let findVirtual verify it
exists with the expected signature, and return nil on ANY failure —
including LambdaMetafactory itself being unavailable, which is what happens
under native-image, where the codec silently keeps its reflection path.
A wrong derivation is never wrong bytes, only a missed optimisation.
raw docstring

clj-protobuf.impl.message

The compiled message: three deftypes implementing protobuf-java's Message, Message.Builder and Parser interfaces over the compiler's slot layout, so that everything downstream — the generated code's .newBuilderForType / .build, core/encode and core/decode, grpc marshallers, TextFormat and JsonFormat — keeps working, with none of DynamicMessage's FieldSet behind it.

A CompiledMessage is a compiled type, an Object[] of slots and an UnknownFieldSet (nil when empty). It is immutable; its serialized size is memoized. A CompiledBuilder is the same, mutable. Building hands the slot array to the message rather than copying it, so a builder that has built, or was made by toBuilder or clone, owns nothing: it copies the array and its collections before its next write. See SlotOwner. The parser owns fresh slots, so parsing mutates freely.

The reflective API returns what protobuf-java's does — EnumValueDescriptor for enums, a list of entry messages for maps, the nested default instance for an unset message field — and equals/hashCode follow AbstractMessage's algorithm exactly, so a compiled message equals and hashes like a DynamicMessage of the same descriptor and value. Descriptors from another pool (a generated class) are never equal, by protobuf-java's own rule.

The compiled message: three deftypes implementing protobuf-java's
Message, Message.Builder and Parser interfaces over the compiler's slot
layout, so that everything downstream — the generated code's
`.newBuilderForType` / `.build`, `core/encode` and `core/decode`, grpc
marshallers, TextFormat and JsonFormat — keeps working, with none of
DynamicMessage's FieldSet behind it.

A CompiledMessage is a compiled type, an Object[] of slots and an
UnknownFieldSet (nil when empty). It is immutable; its serialized size is
memoized. A CompiledBuilder is the same, mutable. Building hands the slot
array to the message rather than copying it, so a builder that has built,
or was made by toBuilder or clone, owns nothing: it copies the array and
its collections before its next write. See SlotOwner. The parser owns
fresh slots, so parsing mutates freely.

The reflective API returns what protobuf-java's does — EnumValueDescriptor
for enums, a list of entry messages for maps, the nested default instance
for an unset message field — and equals/hashCode follow AbstractMessage's
algorithm exactly, so a compiled message equals and hashes like a
DynamicMessage of the same descriptor and value. Descriptors from another
pool (a generated class) are never equal, by protobuf-java's own rule.
raw docstring

clj-protobuf.impl.naming

The one naming rule, shared with the emitter.

protoc-gen-clojure kebab-cases proto field names into record fields and map keys with exactly this algorithm (its field-key-symbol). The runtime's generic nested-map path must produce the same keys byte for byte, or records built by generated code and maps built by the runtime stop being interchangeable. Any change here is a wire-compatibility break with every generated file in existence — don't.

Kebab-casing is lossy (STYLE_LEGACY files can mix conventions), which is why the emitted rt/field lookups carry the exact proto name and this fn is used only for the Clojure-side keys.

The one naming rule, shared with the emitter.

protoc-gen-clojure kebab-cases proto field names into record fields and map
keys with exactly this algorithm (its `field-key-symbol`). The runtime's
generic nested-map path must produce the same keys byte for byte, or records
built by generated code and maps built by the runtime stop being
interchangeable. Any change here is a wire-compatibility break with every
generated file in existence — don't.

Kebab-casing is lossy (STYLE_LEGACY files can mix conventions), which is why
the emitted `rt/field` lookups carry the exact proto name and this fn is used
only for the Clojure-side keys.
raw docstring

clj-protobuf.impl.wire

Wire primitives for the compiled codec: one writer and one reader per field, built once from the field's type and number, over protobuf-java's own CodedOutputStream and CodedInputStream. Varints, zigzag, fixed widths, UTF-8 and the length arithmetic stay protobuf-java's code; what this namespace adds is the choice of which call to make, made once per field instead of once per value.

A writer writes one field — tag included, every element for repeateds, one length-delimited run when packed — and reports its serialized size. A reader is handed the input positioned just after this field's tag and the slot's current value, and returns the slot's new value: the scalar, the list with one more element, the map with one more entry, or a merged message when a singular message field repeats on the wire. Readers are keyed by tag rather than by field number upstream, because a repeated scalar may arrive packed or expanded whatever its descriptor says it writes, and the two are different tags.

Slot representations, shared with the compiler and the codec: int32 kinds Integer int64 kinds Long float Float double Double bool Boolean string String bytes ByteString enum Integer (the number) message Message repeated java.util.ArrayList map java.util.LinkedHashMap (insertion order is wire order)

Wire primitives for the compiled codec: one writer and one reader per
field, built once from the field's type and number, over protobuf-java's
own CodedOutputStream and CodedInputStream. Varints, zigzag, fixed widths,
UTF-8 and the length arithmetic stay protobuf-java's code; what this
namespace adds is the choice of which call to make, made once per field
instead of once per value.

A writer writes one field — tag included, every element for repeateds, one
length-delimited run when packed — and reports its serialized size. A
reader is handed the input positioned just after this field's tag and the
slot's current value, and returns the slot's new value: the scalar, the
list with one more element, the map with one more entry, or a merged
message when a singular message field repeats on the wire. Readers are
keyed by tag rather than by field number upstream, because a repeated
scalar may arrive packed or expanded whatever its descriptor says it
writes, and the two are different tags.

Slot representations, shared with the compiler and the codec:
  int32 kinds  Integer      int64 kinds  Long
  float        Float        double       Double
  bool         Boolean      string       String
  bytes        ByteString   enum         Integer (the number)
  message      Message      repeated     java.util.ArrayList
  map          java.util.LinkedHashMap (insertion order is wire order)
raw docstring

clj-protobuf.runtime

The runtime half of protoc-gen-clojure's generated-code contract.

Generated files call exactly four things here: file-descriptor to rebuild their embedded FileDescriptorProto, known-file for well-known-type deps, message for a prototype per message, and field for a handle per field. Everything protobuf decides per edition — presence, delimited encoding, utf8 validation, packedness — is resolved by protobuf-java when the descriptor is built, which is why generated code never mentions editions at all.

field returns a precomputed FieldHandle rather than a bare FieldDescriptor: the codec's hot path dispatches on a keyword and never touches the descriptor API per call. Handles are built against a specific prototype, so a message-typed field's nested prototype has the right concrete class in both the DynamicMessage and generated-class arms.

The runtime half of protoc-gen-clojure's generated-code contract.

Generated files call exactly four things here: `file-descriptor` to rebuild
their embedded FileDescriptorProto, `known-file` for well-known-type deps,
`message` for a prototype per message, and `field` for a handle per field.
Everything protobuf decides per edition — presence, delimited encoding, utf8
validation, packedness — is resolved by protobuf-java when the descriptor is
built, which is why generated code never mentions editions at all.

`field` returns a precomputed FieldHandle rather than a bare FieldDescriptor:
the codec's hot path dispatches on a keyword and never touches the descriptor
API per call. Handles are built against a specific prototype, so a
message-typed field's nested prototype has the right concrete class in both
the DynamicMessage and generated-class arms.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
Move to previous article
Move to next article
Ctrl+/Jump to the search field
× close