Liking cljdoc? Tell your friends :D

CI License Language Dependencies

Release v0.1.17 — every binding ships this one version; VERSION is the source of truth, scripts/sync-versions.sh propagates it.

pbwire — a dynamic Protobuf runtime for C, no code generation

pbwire is a dependency-free Protobuf runtime written in C. It loads a schema at runtime — a hand-built descriptor tree or a standard FileDescriptorSet — then builds, reads, encodes and decodes messages dynamically through reflection. No protoc step, no generated message code, no libprotobuf in the process. It speaks proto2, proto3 and edition-2023 descriptor sets, the JSON mapping and the text format, and drives twenty-three host-language bindings from that one core.

The point is not to re-implement Protobuf encoding. It is to let every language with a C ABI share one dynamic Protobuf schema and one message-processing core instead of re-implementing Protobuf per host language.

uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len);   /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_free(buf);

Documentation

DocumentContents
C API guidethe tutorial, hand-built schema, FileDescriptorSet load, pbw_ctx loop, ownership rules, streaming, unknown fields, JSON/text, ABI
Design and architecturehow it works, dynamic message semantics, the two allocation strategies, scope, limits, use cases, repository layout
Performance benchmarksthe performance document: the four standard tiers, every binding against its official runtime, and the cross-binding comparison
Performance attempts ledgerevery optimization tried against a still-below-baseline cell, its A/B number, and why it was kept or rejected
Examples indexone runnable round trip per binding, with the exact command
Testing and conformancethe self-check, pinned reference bytes, difftest, the official conformance suite
FAQsyntax support, thread safety, unknown fields, untrusted input, .proto at runtime
Protobuf pain pointswhich official-library complaints pbwire solves, eases or leaves alone — the searchable index
Bindingsevery host binding, where its schema comes from, and the per-language pain point each one removes
include/pbwire.hthe whole public API with per-function contracts
AGENTS.md · CONTRIBUTING.md · llms.txtworking on this repo, commit rules, a one-page factual summary
ROADMAP.mdwhat is done, what is never, what is next

Why use pbwire

  • Schema at runtime. Load a FileDescriptorSet produced by protoc, buf or pbwire-protoc, and handle message types you did not compile in.
  • No runtime dependencies. C99, one public header, ~13k lines of C (about 7.5k of it the binary codec, the rest the JSON mapping and text format). make and make test need nothing but a compiler — no libprotobuf, no protoc, no JSON/C++ runtime underneath.
  • Dynamic messages and reflection. Create messages from a descriptor, get/set fields by number or name, walk nested structures, oneofs and maps.
  • Thin models, not fat codecs. One C core owns every wire byte; codegen emits only field definitions (names, numbers, types, presence) for the host language. Extra annotations, attributes or comments on the model never touch the codec, so the output stays readable, editable and fast to compile.
  • Byte-identical to the reference implementation. Encodings match protoc 36.2 for every scalar type, packed/unpacked repetition and nested messages; pinned bytes in the test suite keep parity from regressing silently.
  • Built for embedding. Stable C ABI, no globals, caller-confined schema/message objects, arena-backed pbw_ctx for per-thread hot loops, wire input validated at the boundary. bindings/embedded packages that as one pbwe_box: one arena (on the heap, or in caller memory via pbwe_open_mem/pbw_ctx_new_mem, which never allocates), caller-owned encode buffers (bare or length-prefixed frames), and both inputs (frames and the startup descriptor set) capped by pbw_limits; make embedded-bench measures it against nanopb, byte for byte and ns for ns.
  • One core, twenty-three host languages. C, C++, Python, Ruby, Java, Kotlin, Node, Objective-C, PHP, Dart, C#, Rust, Go, Erlang, Lua, Swift, Scala, Elixir, Zig, F#, Clojure, Julia and R all speak the same wire format, each with a measured answer to its runtime's own pain point — see what each binding solves.

At a glance

LanguageC99 (C++ consumers include the same header)
Wire formatGoogle Protobuf binary, proto2 + proto3 (edition-2023 descriptor sets load too)
Schema inputhand-built descriptors, or a standard FileDescriptorSet
Message handlingdynamic value tree with reflection; no generated codec code (optional thin models carry field definitions only)
Runtime dependenciesnone (optional importer pbwire-protoc needs libprotobuf, C++17)
Not implemented.proto text parsing in the core (deliberately never: build the set once with protoc/buf/pbwire-protoc); RPC transport and stubs (services load, are queryable, and resolve to descriptors with pbw_service_resolve for routing — the transport itself stays official gRPC/Twirp/plain HTTP); no CLI (inspect with protoc --decode/--encode)
Memory modelone allocation per decoded tree, or one reusable arena pbw_ctx per thread
Limitsnesting depth 256, field numbers up to 2^29-1
PlatformsLinux, macOS, Windows (MSVC through CMake/ctest); any C99 compiler; 32-bit (ILP32) and big-endian are CI-verified
LicenseApache-2.0

Requirements and build

A C99 compiler for the core. CMake plus a C++17 compiler and libprotobuf only for pbwire-protoc.

make          # build build/libpbwire.a
make test     # self-checks: core, threads, descriptor set, C structs, examples round trip
make shared   # libpbwire.so.1 / libpbwire.1.dylib, for FFI hosts that dlopen it
make format   # clang-format (>= 19); make format-check
make clean

CMake is the packaging path and the only build that knows about pbwire-protoc:

cmake -B build-cmake                        # core only, no protobuf needed
cmake -B build-cmake -DPBW_BUILD_PROTOC=ON   # core + .proto importer (needs libprotobuf)
cmake --build build-cmake
(cd build-cmake && ctest)                    # same checks as `make test`, plus test_importer

# installed targets: pbwire::pbwire and pbwire::protoc
cmake --install build-cmake --prefix /usr/local

# then, in a consumer:
# find_package(pbwire REQUIRED)
# target_link_libraries(app PRIVATE pbwire::pbwire)

Everything else is optional and outside the commit gate: make difftest, make conformance and the per-binding *-test / *-bench targets. See docs/testing.md for what they verify and when they are worth running; binding targets are listed in each binding's README and in BENCHMARKS.md.

Quick start

#include "pbwire.h"

pbw_schema *schema = pbw_schema_new();
pbw_descriptor *person = pbw_schema_add_message(schema, "Person");
pbw_message_add_field(person, "name", 1, PBW_TYPE_STRING, PBW_LABEL_OPTIONAL);
pbw_message_add_field(person, "id", 2, PBW_TYPE_INT32, PBW_LABEL_OPTIONAL);
pbw_schema_link(schema);

pbw_message *msg = pbw_message_new(person);
pbw_message_set(msg, 1, pbw_value_new_string("abc"));
pbw_message_set(msg, 2, pbw_value_new_int(-1));

uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len);   /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */

pbw_message *decoded;
pbw_decode(person, buf, len, &decoded);
const char *name = pbw_value_get_string(pbw_message_get(decoded, 1));

pbw_free(buf);
pbw_message_free(decoded);
pbw_message_free(msg);
pbw_schema_free(schema);

Real programs rarely build a schema by hand — they load a FileDescriptorSet at runtime and process types they were never compiled against. The walk-through for that, the arena-backed hot loop, streaming framing and the JSON/text entry points is docs/guide.md; a runnable tutorial schema (proto2 address book, loaded from addressbook.desc at runtime) is in examples/ and runs with make examples.

Performance

make bench verifies byte-for-byte equality with libprotobuf before timing anything, so a speed number can never come from a wrong encoder. On an Apple M4 Pro with -O2 and protoc 36.2, the in-process dynamic core with a pbw_ctx runs 1.24-1.56x the throughput of the generated C++ codec on encode and 1.50-1.81x on decode, on every tier (132 B to 86 KB), and 11.7-18.0x the speed of C++ DynamicMessage on encode and 3.0-5.3x on decode. Generated code only wins where a dynamic codec meets generated straight-line writes on a tiny message (Java/Kotlin/Clojure, 136 B, 0.77x/0.83x/0.74x); the arena and plain APIs win every tier. make bench-docker runs the same harnesses in a Linux container and make bench-compare pairs the two runs row by row, so the numbers above hold off Apple silicon as well (BENCHMARKS.md).

The per-language tables — the same four payload tiers for every binding against its official runtime (Python, Java, Kotlin, Node, Ruby, Objective-C, PHP, Dart, Rust, C#, Erlang, Swift, Scala, Elixir, F#, Clojure, Julia), plus the fastest third-party runtime wherever one meaningfully exists (Node, Rust, C#, PHP), the C struct codec, 1-vs-8-thread scaling, and the reason behind each gap — live in BENCHMARKS.md. The one-row summary is here; the detail is there.

LanguageHow it talks to the coreBest encode / decode vs officialBaseline
Cstatic link, struct codec1.29x (xlarge) / 2.02x (xlarge)protoc C++
C++same C runtime, same header1.29x (xlarge) / 2.02x (xlarge)¹protoc C++
PythonCPython extension6.87x (xlarge) / 1.19x (xlarge) (64x via encode(parsed))google.protobuf (upb)
RubyC extension, linked1.35x (small) / 1.86x (large)google-protobuf gem
Javapure-Java codec, no native code1.23x (medium) / 1.95x (xlarge)protobuf-java
KotlinJava runtime + DSL1.13x (medium) / 1.68x (xlarge)protobuf-kotlin
NodeN-API addon3.18x (xlarge) / 3.17x (xlarge) vs protobufjsjspb + protobufjs
Objective-CARC wrapper7.44x (small) / 2.83x (medium)protobuf-objc
PHPC extension1.70x (small) / 1.90x (small) vs C ext (373x / 135x vs pure PHP)protobuf-php
Dartdart:ffi13.7x (xlarge) / 4.4x (xlarge)protobuf-dart
Ruststatic link1.29x (small) / 1.86x (xlarge) vs prostrust-protobuf 3.x + prost
C#P/Invoke3.19x (medium) / 2.08x (medium)Google.Protobuf + protobuf-net
Gocgo dynamic (hot-swappable registry)2.77x (xlarge) / 2.31x (xlarge)google.golang.org/protobuf
ErlangNIF3.91x (xlarge) / 2.47x (medium)gpb
LuaC module, tables + builder8.93x (xlarge, builder) / 1.66x (small, tables)lua-protobuf
SwiftSwiftPM, static link5.51x (xlarge) / 4.24x (medium)swift-protobuf
ScalaJava runtime + DSL1.48x (medium) / 1.21x (xlarge)ScalaPB
ElixirNIF (shared core)3.06x (small) / 1.90x (xlarge)elixir-protobuf
Zigstatic link1.87x (xlarge) / 50.0x (small)zig-protobuf
F#C# runtime + facade3.19x (xlarge) / 1.99x (medium)Google.Protobuf
ClojureJava runtime + interop1.13x (large) / 1.66x (xlarge)protobuf-java
Juliadlopen + ccall133.5x (xlarge) / 1.31x (medium, workspace decode)ProtoBuf.jl
RC extension, named lists59.0x (large) / 50.3x (xlarge)RProtoBuf

¹ C++ has no separate bench; same engine as the C struct codec. Best columns take the best ratio across the four tiers for encode/decode separately, tier in parentheses (Apple M4 Pro, protoc 36.2); tiers and absolute MB/s are in BENCHMARKS.md.

Comparison with generated code, DynamicMessage and upb

Schema at runtimeGenerated codeRuntime dependencyDynamic messages + reflection
pbwireyesnonenone (C99)yes
protoc generated C++noyes (per-message codec, slow to compile, hard to read/edit)libprotobufvia DynamicMessage, 11-15x slower encode than pbwire ctx
DynamicMessage (C++)yesnolibprotobufyes
upbyesoptionalbundled C runtimeyes
nanopbnoyesnone (embedded C)no

Pick pbwire when the schema is only known at runtime, when one C core serves several host languages, or when libprotobuf must not be in the process. Pick generated code only when the schema never changes and you accept a codegen step: even then generated straight-line writes win only the smallest encode cells against dynamic dispatch — the Java, Kotlin and Clojure bindings at 136 B (0.77x, 0.83x and 0.74x) — while the in-process pbw_ctx wins every tier of the core benchmark and the C struct codec wins every tier of its own.

Contributing

See CONTRIBUTING.md. Enable hooks once per clone:

./scripts/setup-git-hooks.sh

License

Apache-2.0.

Can you improve this documentation?Edit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close