Release
v0.1.17— every binding ships this one version;VERSIONis the source of truth,scripts/sync-versions.shpropagates it.
pbwire is a dependency-free Protobuf runtime written in C. It loads a schema
at runtime — a hand-built descriptor tree or a standard FileDescriptorSet —
then builds, reads, encodes and decodes messages dynamically through reflection.
No protoc step, no generated message code, no libprotobuf in the process. It
speaks proto2, proto3 and edition-2023 descriptor sets, the JSON mapping and the
text format, and drives twenty-three host-language bindings from that one core.
The point is not to re-implement Protobuf encoding. It is to let every language with a C ABI share one dynamic Protobuf schema and one message-processing core instead of re-implementing Protobuf per host language.
uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len); /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_free(buf);
| Document | Contents |
|---|---|
| C API guide | the tutorial, hand-built schema, FileDescriptorSet load, pbw_ctx loop, ownership rules, streaming, unknown fields, JSON/text, ABI |
| Design and architecture | how it works, dynamic message semantics, the two allocation strategies, scope, limits, use cases, repository layout |
| Performance benchmarks | the performance document: the four standard tiers, every binding against its official runtime, and the cross-binding comparison |
| Performance attempts ledger | every optimization tried against a still-below-baseline cell, its A/B number, and why it was kept or rejected |
| Examples index | one runnable round trip per binding, with the exact command |
| Testing and conformance | the self-check, pinned reference bytes, difftest, the official conformance suite |
| FAQ | syntax support, thread safety, unknown fields, untrusted input, .proto at runtime |
| Protobuf pain points | which official-library complaints pbwire solves, eases or leaves alone — the searchable index |
| Bindings | every host binding, where its schema comes from, and the per-language pain point each one removes |
| include/pbwire.h | the whole public API with per-function contracts |
| AGENTS.md · CONTRIBUTING.md · llms.txt | working on this repo, commit rules, a one-page factual summary |
| ROADMAP.md | what is done, what is never, what is next |
FileDescriptorSet produced by protoc, buf
or pbwire-protoc, and handle message types you did not compile in.make and make test need nothing but a compiler — no libprotobuf, no
protoc, no JSON/C++ runtime underneath.codegen
emits only field definitions (names, numbers, types, presence) for the host
language. Extra annotations, attributes or comments on the model never touch
the codec, so the output stays readable, editable and fast to compile.protoc 36.2 for every scalar type, packed/unpacked repetition and nested
messages; pinned bytes in the test suite keep parity from regressing silently.pbw_ctx for per-thread hot loops, wire
input validated at the boundary. bindings/embedded
packages that as one pbwe_box: one arena (on the heap, or in caller memory
via pbwe_open_mem/pbw_ctx_new_mem, which never allocates), caller-owned
encode buffers (bare or length-prefixed frames), and both inputs (frames and
the startup descriptor set) capped by pbw_limits; make embedded-bench
measures it against nanopb, byte for byte and ns for ns.| Language | C99 (C++ consumers include the same header) |
| Wire format | Google Protobuf binary, proto2 + proto3 (edition-2023 descriptor sets load too) |
| Schema input | hand-built descriptors, or a standard FileDescriptorSet |
| Message handling | dynamic value tree with reflection; no generated codec code (optional thin models carry field definitions only) |
| Runtime dependencies | none (optional importer pbwire-protoc needs libprotobuf, C++17) |
| Not implemented | .proto text parsing in the core (deliberately never: build the set once with protoc/buf/pbwire-protoc); RPC transport and stubs (services load, are queryable, and resolve to descriptors with pbw_service_resolve for routing — the transport itself stays official gRPC/Twirp/plain HTTP); no CLI (inspect with protoc --decode/--encode) |
| Memory model | one allocation per decoded tree, or one reusable arena pbw_ctx per thread |
| Limits | nesting depth 256, field numbers up to 2^29-1 |
| Platforms | Linux, macOS, Windows (MSVC through CMake/ctest); any C99 compiler; 32-bit (ILP32) and big-endian are CI-verified |
| License | Apache-2.0 |
A C99 compiler for the core. CMake plus a C++17 compiler and libprotobuf only
for pbwire-protoc.
make # build build/libpbwire.a
make test # self-checks: core, threads, descriptor set, C structs, examples round trip
make shared # libpbwire.so.1 / libpbwire.1.dylib, for FFI hosts that dlopen it
make format # clang-format (>= 19); make format-check
make clean
CMake is the packaging path and the only build that knows about
pbwire-protoc:
cmake -B build-cmake # core only, no protobuf needed
cmake -B build-cmake -DPBW_BUILD_PROTOC=ON # core + .proto importer (needs libprotobuf)
cmake --build build-cmake
(cd build-cmake && ctest) # same checks as `make test`, plus test_importer
# installed targets: pbwire::pbwire and pbwire::protoc
cmake --install build-cmake --prefix /usr/local
# then, in a consumer:
# find_package(pbwire REQUIRED)
# target_link_libraries(app PRIVATE pbwire::pbwire)
Everything else is optional and outside the commit gate: make difftest,
make conformance and the per-binding *-test / *-bench targets. See
docs/testing.md for what they verify and when they are worth
running; binding targets are listed in each binding's README and in
BENCHMARKS.md.
#include "pbwire.h"
pbw_schema *schema = pbw_schema_new();
pbw_descriptor *person = pbw_schema_add_message(schema, "Person");
pbw_message_add_field(person, "name", 1, PBW_TYPE_STRING, PBW_LABEL_OPTIONAL);
pbw_message_add_field(person, "id", 2, PBW_TYPE_INT32, PBW_LABEL_OPTIONAL);
pbw_schema_link(schema);
pbw_message *msg = pbw_message_new(person);
pbw_message_set(msg, 1, pbw_value_new_string("abc"));
pbw_message_set(msg, 2, pbw_value_new_int(-1));
uint8_t *buf;
size_t len;
pbw_encode(msg, &buf, &len); /* 0a 03 61 62 63 10 ff ff ff ff ff ff ff ff ff 01 */
pbw_message *decoded;
pbw_decode(person, buf, len, &decoded);
const char *name = pbw_value_get_string(pbw_message_get(decoded, 1));
pbw_free(buf);
pbw_message_free(decoded);
pbw_message_free(msg);
pbw_schema_free(schema);
Real programs rarely build a schema by hand — they load a FileDescriptorSet
at runtime and process types they were never compiled against. The walk-through
for that, the arena-backed hot loop, streaming framing and the JSON/text entry
points is docs/guide.md; a runnable tutorial schema (proto2
address book, loaded from addressbook.desc at runtime) is in
examples/ and runs with make examples.
make bench verifies byte-for-byte equality with libprotobuf before timing
anything, so a speed number can never come from a wrong encoder. On an Apple
M4 Pro with -O2 and protoc 36.2, the in-process dynamic core with a
pbw_ctx runs 1.24-1.56x the throughput of the generated C++ codec on encode
and 1.50-1.81x on decode, on every tier
(132 B to 86 KB), and 11.7-18.0x the speed of C++
DynamicMessage on encode and 3.0-5.3x on decode. Generated code only wins where
a dynamic codec meets generated straight-line writes on a tiny message
(Java/Kotlin/Clojure, 136 B, 0.77x/0.83x/0.74x); the arena and plain APIs win
every tier. make bench-docker runs the
same harnesses in a Linux container and make bench-compare pairs the two runs
row by row, so the numbers above hold off Apple silicon as well
(BENCHMARKS.md).
The per-language tables — the same four payload tiers for every binding against its official runtime (Python, Java, Kotlin, Node, Ruby, Objective-C, PHP, Dart, Rust, C#, Erlang, Swift, Scala, Elixir, F#, Clojure, Julia), plus the fastest third-party runtime wherever one meaningfully exists (Node, Rust, C#, PHP), the C struct codec, 1-vs-8-thread scaling, and the reason behind each gap — live in BENCHMARKS.md. The one-row summary is here; the detail is there.
| Language | How it talks to the core | Best encode / decode vs official | Baseline |
|---|---|---|---|
| C | static link, struct codec | 1.29x (xlarge) / 2.02x (xlarge) | protoc C++ |
| C++ | same C runtime, same header | 1.29x (xlarge) / 2.02x (xlarge)¹ | protoc C++ |
| Python | CPython extension | 6.87x (xlarge) / 1.19x (xlarge) (64x via encode(parsed)) | google.protobuf (upb) |
| Ruby | C extension, linked | 1.35x (small) / 1.86x (large) | google-protobuf gem |
| Java | pure-Java codec, no native code | 1.23x (medium) / 1.95x (xlarge) | protobuf-java |
| Kotlin | Java runtime + DSL | 1.13x (medium) / 1.68x (xlarge) | protobuf-kotlin |
| Node | N-API addon | 3.18x (xlarge) / 3.17x (xlarge) vs protobufjs | jspb + protobufjs |
| Objective-C | ARC wrapper | 7.44x (small) / 2.83x (medium) | protobuf-objc |
| PHP | C extension | 1.70x (small) / 1.90x (small) vs C ext (373x / 135x vs pure PHP) | protobuf-php |
| Dart | dart:ffi | 13.7x (xlarge) / 4.4x (xlarge) | protobuf-dart |
| Rust | static link | 1.29x (small) / 1.86x (xlarge) vs prost | rust-protobuf 3.x + prost |
| C# | P/Invoke | 3.19x (medium) / 2.08x (medium) | Google.Protobuf + protobuf-net |
| Go | cgo dynamic (hot-swappable registry) | 2.77x (xlarge) / 2.31x (xlarge) | google.golang.org/protobuf |
| Erlang | NIF | 3.91x (xlarge) / 2.47x (medium) | gpb |
| Lua | C module, tables + builder | 8.93x (xlarge, builder) / 1.66x (small, tables) | lua-protobuf |
| Swift | SwiftPM, static link | 5.51x (xlarge) / 4.24x (medium) | swift-protobuf |
| Scala | Java runtime + DSL | 1.48x (medium) / 1.21x (xlarge) | ScalaPB |
| Elixir | NIF (shared core) | 3.06x (small) / 1.90x (xlarge) | elixir-protobuf |
| Zig | static link | 1.87x (xlarge) / 50.0x (small) | zig-protobuf |
| F# | C# runtime + facade | 3.19x (xlarge) / 1.99x (medium) | Google.Protobuf |
| Clojure | Java runtime + interop | 1.13x (large) / 1.66x (xlarge) | protobuf-java |
| Julia | dlopen + ccall | 133.5x (xlarge) / 1.31x (medium, workspace decode) | ProtoBuf.jl |
| R | C extension, named lists | 59.0x (large) / 50.3x (xlarge) | RProtoBuf |
¹ C++ has no separate bench; same engine as the C struct codec. Best columns take the best ratio across the four tiers for encode/decode separately, tier in parentheses (Apple M4 Pro, protoc 36.2); tiers and absolute MB/s are in BENCHMARKS.md.
DynamicMessage and upb| Schema at runtime | Generated code | Runtime dependency | Dynamic messages + reflection | |
|---|---|---|---|---|
| pbwire | yes | none | none (C99) | yes |
protoc generated C++ | no | yes (per-message codec, slow to compile, hard to read/edit) | libprotobuf | via DynamicMessage, 11-15x slower encode than pbwire ctx |
DynamicMessage (C++) | yes | no | libprotobuf | yes |
| upb | yes | optional | bundled C runtime | yes |
| nanopb | no | yes | none (embedded C) | no |
Pick pbwire when the schema is only known at runtime, when one C core serves
several host languages, or when libprotobuf must not be in the process. Pick
generated code only when the schema never changes and you accept a codegen
step: even then generated straight-line writes win only the smallest encode
cells against dynamic dispatch — the Java, Kotlin and Clojure bindings at 136 B
(0.77x, 0.83x and 0.74x) — while the in-process pbw_ctx wins every tier of the
core benchmark and the C struct codec wins every tier of its own.
See CONTRIBUTING.md. Enable hooks once per clone:
./scripts/setup-git-hooks.sh
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |