Liking cljdoc? Tell your friends :D

pbwire FAQ

Is pbwire a Protobuf code generator?

No codec code is ever generated. It is a runtime: descriptors in, dynamic messages out. Nothing is compiled per schema, and adding a message type does not require a rebuild — unless you embedded the descriptor set in the binary (see Do I have to recompile when a .proto changes?). The optional codegen layer emits thin models only — field definitions (names, numbers, types, presence) for the host compiler to check — while every byte still goes through the one shared codec. Custom annotations or attributes on those models never affect the wire bytes.

Does it need protoc at runtime?

No. protoc (or buf, or the optional pbwire-protoc library) produces a FileDescriptorSet once; pbwire reads those bytes at runtime.

Which Protobuf syntaxes are supported?

proto2 and proto3, including edition-2023 descriptor sets (editions resolve their FeatureSet per file/message/field: packed default, required, closed enums, delimited messages). Groups decode into their submessage like message fields, including required checks inside them. Extensions attach to their extendee (pbw_field_is_extension() tells them apart; MessageSet items decode into their extension and encode back byte-identically). A proto2 closed enum's unknown number is kept as an unknown field, and pbw_decode refuses a message missing a required field.

Does pbwire support JSON or text format?

Text format yes (pbw_text_print / pbw_text_parse, byte-identical printer output to the reference, strict parser parity including extensions, groups, Any and unknown fields). JSON yes (pbw_json_print / pbw_json_parse, same verdicts as the reference parser on 40,000 mutated inputs, all well-known types, extensions as [full.name], strict base64 and number rules). See guide.md.

Can I use it from another language?

Yes — that is the intended use. All twenty-three bindings live under bindings/, each with a round-trip test, a benchmark against its official runtime and, where it applies, the conformance testee. The per-language pain point each one removes is the table in bindings/README.md; the short version:

languagehow it reaches the coreschema
Cdirect callFileDescriptorSet, plus plugin/ layout tables
C++direct call over the C runtimeFileDescriptorSet
PythonCPython extension, dict↔wire in CFileDescriptorSet
RubyC extension linked against the coreFileDescriptorSet
Javaown pure-Java codec (no JNI, no native library)FileDescriptorSet
Kotlinthe Java runtime + a DSLFileDescriptorSet
Nodehand-written N-API addon (no npm)FileDescriptorSet
Objective-CARC wrapper (Foundation only)FileDescriptorSet
PHPnative C extension (ext-ffi wrapper without a compiler)FileDescriptorSet
Dartdart:ffiFileDescriptorSet
C#DllImport + a resolverFileDescriptorSet
Rustlinks libpbwire.a staticallyFileDescriptorSet
Gocgo dynamic binding over the C coreFileDescriptorSet (hot-swappable)
ErlangNIF; gpb records via pbwire:load_cache/1gpb's get_msg_defs/0
LuaC module over the core, tables in/outFileDescriptorSet
SwiftSwiftPM package, FFI host over the C ABIFileDescriptorSet
Scalathe Java runtime + a DSLFileDescriptorSet
Elixirstruct translation over the shared NIFelixir-protobuf structs
Zigstatic link, core sources compiled inFileDescriptorSet
F#facade over the C# projectFileDescriptorSet
Clojurepersistent maps over the Java runtimeFileDescriptorSet
Juliadlopen + ccall FFI hostFileDescriptorSet
RC extension, named lists in/outFileDescriptorSet

The C bindings check PBW_ABI_VERSION at load time, so a library built from a newer, incompatible tree fails fast instead of corrupting memory. The C ABI is the product; richer host bindings live on top of it. codegen adds generated message models per language: all but Java ride the same dynamic API (Java emits a direct Wire codec).

When should I not use pbwire?

When the schema is fixed at build time, you never touch a descriptor, and libprotobuf is already in the process: then generated code is the simpler choice, with fewer moving parts and no runtime schema to ship. pbwire still wins the throughput comparison even there (the in-process core is 1.1-1.5x the generated C++ codec), but throughput is not the only axis — a straight-line codec has no schema to load and no reflection work to do. pbwire also does not do RPC. See ../README.md and design.md.

Is it thread-safe?

There is no global state. A schema or message must be confined to one thread by the caller, and one pbw_ctx per thread keeps arena reuse local. No locks, no thread-local buffers.

Does it work on 32-bit and big-endian hosts?

Yes, and CI proves it: the whole gate runs with -m32 (ILP32) on an x86_64 runner, and the dependency-free suites run on s390x under qemu-user for big-endian. Both jobs exist because they found real bugs the little-endian 64-bit default hides — a tag word stored with a 32-bit write, floats read from the wrong half of a 64-bit bit pattern, and fixed-width struct members copied through a uint64_t. See testing.md.

What happens to fields the schema does not declare?

They are preserved as raw wire bytes and re-emitted after the known fields, in input order. A gateway can forward messages for a schema it only partially knows. Unknown fields are why 7 conformance tests pass that would fail otherwise.

Why does it walk a descriptor instead of compiling field offsets in?

Generated code compiles field offsets into straight-line C++; pbwire walks a descriptor and a value tree at runtime, which costs roughly 40 ns per encode of a 132-byte message. That cost is mostly paid back by writing the message in one pass (the C++ serializer sizes first, then writes) and by precomputed tag bytes, so the in-process pbw_ctx API is 1.12-1.47x the generated codec on encode and 1.52-1.82x on decode, on every tier. The gap only shows up in one place: a dynamic codec meeting generated straight-line writes on a tiny message (Java/Kotlin/Clojure, 0.71x/0.78x/0.75x at 136 B). Numbers and the per-language reasons are in BENCHMARKS.md.

Can I decode untrusted bytes with it?

Yes: lengths are bounds-checked, field number 0 and wire types above 5 are rejected, nesting is capped at 256 (PBW_MAX_DEPTH), and every failure returns a pbw_status instead of aborting. pbw_limits lowers the caps per call (max_bytes, max_depth) for stricter hosts. Descriptor sets stay trusted input, but pbw_descriptor_set_load_limits() applies the same two caps to the set itself (oversized/hostile sets fail with PBW_ERR_LIMIT) for hosts that accept sets over the network — see design.md.

How do I load a .proto file at runtime?

Convert it to a FileDescriptorSet once (protoc --include_imports --descriptor_set_out, buf build -o, or the optional pbwire-protoc library), then call pbw_descriptor_set_load() and pbw_schema_link() is already done for you. All imported files must be in the set. See guide.md.

The core will never parse .proto text directly — that is a permanent decision, not a gap: the FileDescriptorSet is the schema format. For inspection and debugging use the official CLI (protoc --decode, protoc --encode); no pbwire CLI is planned.

Do I have to recompile when a .proto changes?

Only if you baked the schema into the binary. The loader takes bytes (pbw_descriptor_set_load), so a sidecar file and embedded bytes are the same call — the difference is deployment, not API:

  • External file (default). Ship the new FileDescriptorSet next to the binary and reload the schema in the running process; no rebuild. Messages of the old schema stay valid until you free that schema.
  • Embedded bytes (single binary). Compile the set in (xxd -i, Rust include_bytes!, app resources) and load it from memory (from_bytes / fromBytes / load(bytes), depending on the binding). No sidecar file to lose, but a .proto change means rebuilding and redeploying.

Exceptions: Erlang uses gpb's compiled definitions and recompiles by construction. The C struct binding still validates its generated layout against a descriptor set at run time, so embed that set too rather than dropping it.

Where do I report a bug?

A minimal reproduction: the bytes (or .proto), the expected behavior, and whether it came from make difftest / make conformance. See CONTRIBUTING.md.

Can you improve this documentation?Edit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close