protoc at runtime?.proto file at runtime?.proto changes?No codec code is ever generated. It is a runtime: descriptors in, dynamic
messages out. Nothing is compiled per schema, and adding a message type does
not require a rebuild — unless you embedded the descriptor set in the binary
(see Do I have to recompile when a .proto changes?).
The optional codegen layer emits thin models only — field definitions
(names, numbers, types, presence) for the host compiler to check — while
every byte still goes through the one shared codec. Custom annotations or
attributes on those models never affect the wire bytes.
protoc at runtime?No. protoc (or buf, or the optional pbwire-protoc library) produces a
FileDescriptorSet once; pbwire reads those bytes at runtime.
proto2 and proto3, including edition-2023 descriptor sets (editions resolve
their FeatureSet per file/message/field: packed default, required, closed
enums, delimited messages). Groups decode into their submessage like message fields, including
required checks inside them. Extensions attach to their extendee
(pbw_field_is_extension() tells them apart; MessageSet items decode into
their extension and encode back byte-identically). A proto2 closed enum's unknown number is kept as an
unknown field, and pbw_decode refuses a message missing a required field.
Text format yes (pbw_text_print / pbw_text_parse, byte-identical printer
output to the reference, strict parser parity including extensions, groups,
Any and unknown fields). JSON yes (pbw_json_print / pbw_json_parse, same
verdicts as the reference parser on 40,000 mutated inputs, all well-known types,
extensions as [full.name], strict base64 and number rules). See
guide.md.
Yes — that is the intended use. All twenty-three bindings live under bindings/, each with a round-trip
test, a benchmark against its official runtime and, where it applies, the
conformance testee. The per-language pain point each one removes is the
table in bindings/README.md;
the short version:
| language | how it reaches the core | schema |
|---|---|---|
| C | direct call | FileDescriptorSet, plus plugin/ layout tables |
| C++ | direct call over the C runtime | FileDescriptorSet |
| Python | CPython extension, dict↔wire in C | FileDescriptorSet |
| Ruby | C extension linked against the core | FileDescriptorSet |
| Java | own pure-Java codec (no JNI, no native library) | FileDescriptorSet |
| Kotlin | the Java runtime + a DSL | FileDescriptorSet |
| Node | hand-written N-API addon (no npm) | FileDescriptorSet |
| Objective-C | ARC wrapper (Foundation only) | FileDescriptorSet |
| PHP | native C extension (ext-ffi wrapper without a compiler) | FileDescriptorSet |
| Dart | dart:ffi | FileDescriptorSet |
| C# | DllImport + a resolver | FileDescriptorSet |
| Rust | links libpbwire.a statically | FileDescriptorSet |
| Go | cgo dynamic binding over the C core | FileDescriptorSet (hot-swappable) |
| Erlang | NIF; gpb records via pbwire:load_cache/1 | gpb's get_msg_defs/0 |
| Lua | C module over the core, tables in/out | FileDescriptorSet |
| Swift | SwiftPM package, FFI host over the C ABI | FileDescriptorSet |
| Scala | the Java runtime + a DSL | FileDescriptorSet |
| Elixir | struct translation over the shared NIF | elixir-protobuf structs |
| Zig | static link, core sources compiled in | FileDescriptorSet |
| F# | facade over the C# project | FileDescriptorSet |
| Clojure | persistent maps over the Java runtime | FileDescriptorSet |
| Julia | dlopen + ccall FFI host | FileDescriptorSet |
| R | C extension, named lists in/out | FileDescriptorSet |
The C bindings check PBW_ABI_VERSION at load time, so a library built from a
newer, incompatible tree fails fast instead of corrupting memory. The C ABI is
the product; richer host bindings live on top of it. codegen adds generated
message models per language: all but Java ride the same dynamic API (Java
emits a direct Wire codec).
When the schema is fixed at build time, you never touch a descriptor, and libprotobuf is already in the process: then generated code is the simpler choice, with fewer moving parts and no runtime schema to ship. pbwire still wins the throughput comparison even there (the in-process core is 1.1-1.5x the generated C++ codec), but throughput is not the only axis — a straight-line codec has no schema to load and no reflection work to do. pbwire also does not do RPC. See ../README.md and design.md.
There is no global state. A schema or message must be confined to one thread by
the caller, and one pbw_ctx per thread keeps arena reuse local. No locks, no
thread-local buffers.
Yes, and CI proves it: the whole gate runs with -m32 (ILP32) on an x86_64
runner, and the dependency-free suites run on s390x under qemu-user for
big-endian. Both jobs exist because they found real bugs the little-endian
64-bit default hides — a tag word stored with a 32-bit write, floats read
from the wrong half of a 64-bit bit pattern, and fixed-width struct members
copied through a uint64_t. See
testing.md.
They are preserved as raw wire bytes and re-emitted after the known fields, in input order. A gateway can forward messages for a schema it only partially knows. Unknown fields are why 7 conformance tests pass that would fail otherwise.
Generated code compiles field offsets into straight-line C++; pbwire walks a
descriptor and a value tree at runtime, which costs roughly 40 ns per encode of a
132-byte message. That cost is mostly paid back by writing the message in one
pass (the C++ serializer sizes first, then writes) and by precomputed tag bytes,
so the in-process pbw_ctx API is 1.12-1.47x the generated codec on encode
and 1.52-1.82x on decode, on every tier.
The gap only shows up in one place: a dynamic codec meeting generated straight-line writes on
a tiny message (Java/Kotlin/Clojure, 0.71x/0.78x/0.75x at 136 B). Numbers
and the per-language
reasons are in BENCHMARKS.md.
Yes: lengths are bounds-checked, field number 0 and wire types above 5 are
rejected, nesting is capped at 256 (PBW_MAX_DEPTH), and every failure returns a
pbw_status instead of aborting. pbw_limits lowers the caps per call
(max_bytes, max_depth) for stricter hosts. Descriptor sets stay trusted
input, but pbw_descriptor_set_load_limits() applies the same two caps to
the set itself (oversized/hostile sets fail with PBW_ERR_LIMIT) for hosts
that accept sets over the network — see design.md.
.proto file at runtime?Convert it to a FileDescriptorSet once (protoc --include_imports --descriptor_set_out, buf build -o, or the optional pbwire-protoc library),
then call pbw_descriptor_set_load() and pbw_schema_link() is already done for
you. All imported files must be in the set. See guide.md.
The core will never parse .proto text directly — that is a permanent
decision, not a gap: the FileDescriptorSet is the schema format. For
inspection and debugging use the official CLI (protoc --decode,
protoc --encode); no pbwire CLI is planned.
.proto changes?Only if you baked the schema into the binary. The loader takes bytes
(pbw_descriptor_set_load), so a sidecar file and embedded bytes are the same
call — the difference is deployment, not API:
FileDescriptorSet next to the
binary and reload the schema in the running process; no rebuild. Messages of
the old schema stay valid until you free that schema.xxd -i, Rust
include_bytes!, app resources) and load it from memory
(from_bytes / fromBytes / load(bytes), depending on the binding). No
sidecar file to lose, but a .proto change means rebuilding and redeploying.Exceptions: Erlang uses gpb's compiled definitions and recompiles by construction. The C struct binding still validates its generated layout against a descriptor set at run time, so embed that set too rather than dropping it.
A minimal reproduction: the bytes (or .proto), the expected behavior, and
whether it came from make difftest / make conformance. See
CONTRIBUTING.md.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |