pbw_limits bounds, encoded size / encode_to / encode_sink,
visit scanning, WKT helpers (Any pack, Timestamp/Duration millis,
FieldMask paths) and service resolution, each as a thin wrapper over the
same C core (no second codec anywhere). JVM bindings (Java, Kotlin, Scala,
Clojure) parse the descriptor set in pure Java; FFI hosts call the C ABI.make conformance-julia, conformance-scala, conformance-clojure,
conformance-fsharp and conformance-zig: the official suite now runs
through those testees too, so 11 bindings cover binary, JSON and text
(conformance-python and conformance-java stopped skipping JSON/text).Json, Text, Wkt, Limits and ServiceType join the runtime, so
the pure-Java path has the same reach as the C core (2753 binary+JSON and 439
text conformance successes).pbw_json_parse_limits() and pbw_text_parse_limits(): the two text parsers
take the pbw_limits{max_bytes, max_depth} the binary decoder already takes,
checked before any recursion, so a host on a small thread stack can lower the
nesting cap per call instead of trusting PBW_MAX_DEPTH (256 levels cost
~1 MB of C stack under sanitizers). Additive, so PBW_ABI_VERSION stays 1.The R binding's context-generation check no longer dereferences a cleared external pointer.
The JVM Wire.MAX_DEPTH matches the core's PBW_MAX_DEPTH (256): a payload
the core accepts is no longer rejected by Java/Kotlin/Scala/Clojure.
Dart looked up pbw_encode_to but never exposed it; Message.encodeTo( nativeBuffer, cap) now wraps it, and the README tables of seven bindings
document the surface they gained (JSON/text, framing, deterministic, limits,
WKT, services, visit).
Dart's visit saved its callback in one module slot, so a callback that
started another scan cleared the outer one mid-pass; it is save/restore now.
The Zig testee flushes each response (a buffered write deadlocked the
runner) and zig build testee now installs zig-out/bin/testee-zig.
pbw_timestamp_to_millis() / pbw_duration_to_millis() rejected seconds
values that only might overflow: the bound reserved 999 ms regardless of the
fraction's sign, so a legal boundary tree (seconds 9223372036854775, nanos
807000000 == INT64_MAX ms) reported PBW_ERR_LIMIT and failed to
round-trip. Both bounds are now derived from the actual sub-second remainder.
pbw_timestamp_from_millis() additionally refuses a millisecond value whose
borrowed second would put the seconds field below the int64/1000 edge
(Duration keeps the nanos' sign and converts both extremes unchanged); the
vendored Erlang copy and the Java Wkt twins follow.
Input bounds silently degraded in five bindings: a negative max_bytes or
max_depth (or a NaN in Node/R) wrapped into the unsigned C field and
turned a cap into "no cap" instead of failing. Node, Ruby, PHP, R and Dart
now reject a value the C type cannot hold, Swift throws instead of clamping it
to 0, and Go/Rust/C#/Julia gained bounded parse_json/parse_text
counterparts (ParseJSONLimits, parse_json_limits, optional parameters),
which the core already supported.
Java's encoder wrote the raw 64-bit varint for an int32/enum field whose
decoded value was wider than 32 bits (e.g. 0x1FFFFFFFF for -1), producing
08 ff ff ff ff 1f instead of the canonical 10-byte sign-extended form. The
singular path now encodes the int32 interpretation, matching the reference
(two recommended conformance cases, and the same fix covers Kotlin/Scala).
Java's decoder recorded implicit presence from the raw wire value: an
int32/uint32 varint wider than 32 bits that truncates to zero stayed "present",
so a re-encode emitted 08 00 where the reference drops the field. Presence
is now decided on the field's own zero (zeroBits), which also covers the
enum and sint32 cases.
Java decoded a string field with the replacing UTF-8 constructor, so an invalid sequence became U+FFFD and the JSON printer emitted it instead of failing like the C core (and the text printer lost the byte entirely). Strings are now strict-decoded: a held value and its JSON/text print throw.
print_unknown = false did not drop the unknown fields in Julia, Rust and C#:
those wrappers sent false to pbw_text_print() (the "print them" default)
instead of pbw_text_print_opts(..., 0). C# PrintText had the same
inversion behind its printUnknown = false default. All three now always use
the opts entry, and Julia/F# text-conformance went 439 -> 443.
The Java and Scala conformance testees ignored print_unknown_fields, so the
four *UnknownFields_Drop.TextFormatOutput cases printed unknown fields
anyway; both suites now report 443 text successes, like the core.
Java's JSON parser accepted null as a map value; the mapping rejects it
(only a singular field, or a google.protobuf.Value map value, may be null).
The Clojure conformance testee decoded binary input to a map, so a re-encode dropped unknown fields and lost the canonical form; it now uses the binding's mutable-message path, and the suite reports 2761 binary+JSON and 443 text successes for C, Java, Scala, Kotlin's runtime, Clojure and the FFI hosts.
The Python extension's setup.py left wkt.c out of the source list, so a
wheel or sdist built from it failed to import with an undefined pbw_any_pack
symbol (the JSON printer reaches the Any helpers). The make python-bind
path was unaffected, which is why the tests did not catch it.
A JSON nesting-limit hit inside an array was reported as
PBW_ERR_INVALID instead of PBW_ERR_LIMIT, unlike the binary decoder and
the singular-object path.
An Any whose @type does not resolve is now a parse error even with
ignore_unknown_fields: the reference parser (protobuf 36.2) resolves the
type before it consults the option and returns INVALID_ARGUMENT, so the
previous "skip the whole field" behaviour was a deviation.
pbw_message_encoded_size() rejects an inconsistent packed list
(PBW_ERR_INVALID) the same way pbw_encode() does, instead of reporting a
length for a tree the writer refuses.
The locale decimal-separator remap (pbw_strtod_slow()) moved the end offset
backwards when a parse stopped inside a multi-byte separator: for a three-byte
separator "1<sep>5" ended one byte early, so the caller lost the fraction.
The shared library exported the internal helpers declared in src/internal.h
(pbw_arena_*, pbw_buf_*, pbw_varint_read, pbw_strtod_slow, ...) next to
the public API, so an embedder could link against them and their names could
not change without an ABI break. The core now builds with
-fvisibility=hidden and include/pbwire.h marks its declarations default
visible: the dynamic symbol table is exactly what the header declares.
PBW_MAX_DEPTH documents its stack cost: the nesting cap bounds recursion,
not stack (~1.4 KB per level in a release build, ~1 MB under sanitizers), and
test_threads now runs the cap-deep decode/JSON/encode paths on a 2 MB worker
stack, so a frame-bloating change fails the gate instead of a host's thread.
An unlinked descriptor no longer leaves the oneof case table out of the
message block: pbw_message_new() sizes it from the fields themselves when
the link has not run, which is the path the JSON/text printers take. Before
this, setting a oneof member of such a message wrote one word past its
allocation.
A failed pbw_schema_link() followed by another pbw_message_add_field()
no longer leaves stale references behind: the add drops the per-descriptor
lookup tables the sort and the array growth invalidated.
pbw_message_set() on a caller-memory context (pbw_ctx_new_mem) refuses a
heap-owning value, like the heap-grown context always did: the reset releases
the arena, so a stored heap payload could never be freed.
pbw_message_add_field() drops the descriptor lookup tables before it grows
the field array, so a failed pbw_schema_link() followed by an add cannot
leave pbw_descriptor_find_field_name() reading a freed row (reachable when
the name allocation fails right after the row array moved).
The JSON parser refuses a message that only carries a well-known-type name
without the real shape, instead of storing or dereferencing a field that is
not there: google.protobuf.Timestamp and Duration require singular
int64 seconds and int32 nanos (a descriptor with no seconds crashed the
parser), and Any requires a singular string type_url and a singular
bytes value.
Any no longer truncates a type_url at 255 bytes: the URL is stored
verbatim, so a printed Any whose URL is longer than a field-name buffer can
be parsed back.
pbw_struct_arena_new_mem() (bindings/c) checks the caller's block size
before writing the arena and block headers: a block too small for them used
to be refused only after the headers had been written past its end.
pbw_ctx_new_mem(): a pbw_ctx whose handle, arena and head chunk live in
a caller-supplied block, so a decode never allocates. An allocation that
does not fit is PBW_ERR_NOMEM, pbw_ctx_reset rewinds the same block and
pbw_ctx_free releases nothing (the block stays the caller's). The encode
buffer cannot grow either, so such a context pairs with pbw_encode_to();
pbw_encode_ctx() reports PBW_ERR_NOMEM.pbwe_open_mem() in bindings/embedded: the caller-memory context behind
the facade, so the message path of a firmware image never touches the heap
(the descriptor set is still loaded at startup). There is no retain_cap:
the block is the cap, and it survives pbwe_reset and pbwe_close.pbw_struct_arena_new_mem() in bindings/c: the struct codec's decode
arena on caller memory, same contract (a fit miss is PBW_ERR_NOMEM, free
releases nothing), so the direct struct codec decodes without allocating.make embedded-bench / make embedded-size (+ scripts/mcu-size.sh): the
embedded tier against nanopb on one message, four-tier throughput with a
byte-equality assertion and Cortex-M flash for both codecs. BENCHMARKS.md
and bindings/embedded/README.md carry the tables and the RAM model.Timestamp parsing rejects a date or time a real calendar cannot spell
(month 13, hour 25, minute or second 61, February 30, and a leap day in a
common year). The reference implementations reject them too; before this the
epoch-day arithmetic rolled the value over silently.required scalar on decode: the
generated layout now carries a has_ flag for such a field, so a wire that
omits one fails with PBW_ERR_REQUIRED instead of decoding as zero. A layout
with a required scalar and no flag is rejected at codec init.[default=] value.required fields (the presence mask was 64 bits
wide): above that it tracks one byte per field and still rejects a wire that
omits one.PbwireException instead of writing past the array. The
length-delimited reader compares pos + len in long too: a hostile length
near Integer.MAX_VALUE wrapped negative, passed the truncation check and
produced a slice past the array.float/double field as a collection
(the singular bitwise-comparer overload did not even compile), and the
generated models are warning-free.encode_to() tries the caller's buffer first and only walks the tree a
second time to size it when it does not fit: the unconditional sizing walk
had halved encode throughput (0.52x prost on the largest tier).conformance/testee.c reads a JSON/text payload at its exact field length
instead of strlen(), matching the reference testee when the payload carries
a NUL byte.nulls was rejected as if it were null), and the name buffers hold
the 512/1024-byte names the descriptor loader accepts.bindings/erlang/c_src/core is a byte-identical copy of the core again
(make erlang-core-sync refreshes it; the NIF build compares).pbw_message_add_field() indexes field numbers and names as it goes, so
building a very wide message is linear: 8000 fields went from 61 ms of
comparing to 0.5 ms.test_nofail compiles the direct struct codec with the same allocation hooks
and sweeps its error paths too, and CMakeLists.txt builds that variant.make codegen-conformance) in all twenty languages.tests/test_descriptor.c covers the descriptor-set loader's oneof-index
and map-entry rejections with hostile FileDescriptorSet fixtures, and
tests/test_pbwire.c covers the caller-memory context.pbw_encode_delimited_to() and pbw_decode_delimited_ctx[_limits](): the
caller-buffer and pbw_ctx twins of the length-delimited framing, so a
framed stream decodes into a workspace arena with no per-frame malloc and
a frame is written without one either. The declared frame length is now
checked against max_bytes in pbw_decode_delimited_limits() too, before
the body is decoded, matching pbw_istream_next().bindings/embedded: pbwe_encode_delimited, pbwe_decode_delimited and
pbwe_decode_delimited_desc, so the facade covers both the bare and the
length-prefixed form without carrying varint code of its own.SECURITY.md: vulnerability reporting policy and supported versions.NOTICE: third-party attributions (descriptor.proto wire shape, test
baselines).scripts/release-check.sh (make release-check): release dry-run — version
consistency, tag-vs-VERSION match, changelog entry, artifact presence.tests/fuzz_decode.c (make fuzz): stdin-fed decode/encode/visit fuzz
harness; CI runs it on a fixed corpus plus a short deterministic mutation
burst.FeatureSet support in pbw_descriptor_set_load(): file-level
and per-field field_presence / repeated_field_encoding are honored
instead of treating every edition as proto3 defaults.FileDescriptorSet services are loaded and queryable
(pbw_schema_service_*, pbw_service_*); the core still does no RPC.message_set_wire_format decode
group items into their extension fields and re-encode them byte-identically.bindings/MIGRATING.md: per-language migration notes from the official
runtimes.examples/gateway/: proto3 + JSON + unknown-field passthrough tutorial.pbw_descriptor_set_load_limits(): the pbw_limits byte/nesting caps applied
to the descriptor set itself (PBW_ERR_LIMIT on oversize/hostile sets), plus
a tests/data/fuzz-desc/ corpus replayed by make fuzz through it.pbw_service_resolve(): service + method name to input/output descriptors,
so a transport can route without the core doing RPC.pbw_timestamp/duration_from_millis,
pbw_fieldmask_path_count/at (the to_millis side already existed).tests/data/fuzz-json/ + tests/data/fuzz-text/ corpora: make fuzz replays
the JSON mapping and text format through the same harness as the wire bytes.scripts/check-llms.sh (+ CI docs-sync job): llms.txt must name every
public pbw_* function, so the AI-retrieval summary cannot drift from the API.test_threads under ThreadSanitizer; nightly runs every ubuntu-capable
binding self-check; release artifacts carry SLSA provenance attestations.
New binding-asan job: the Python/Ruby/Node extensions build instrumented
and run their suites under ASan (LEAK detection stays on for Python; the
Ruby/Node interpreters leak on their own, so they run ASan-only).PBW_EXTRA_CFLAGS hook in the Ruby/Node build.sh drivers (empty by
default; the ASan job passes -fsanitize=address through it).Schema <-> MessageType references are GC-tracked
now (dropping a schema in a loop holds RSS flat; ASan/LSan clean).pbw_schema_error() after
pbw_schema_free() on the load-failure path (found by the ASan job).GenericCodec/PbCodec.setRetainCap(bytes) caps the per-thread
encode buffer (1 MiB default, 0 keeps nothing); Kotlin/Scala/Clojure
inherit it through the shared runtime.pbwire:set_opts([{retain_cap, Bytes}]) caps every scheduler
context (existing and future, under the cache lock); Elixir passes
retain_cap: through Pbwire.load_schema/2 (needs Hex :pbwire past
this release to take effect).binding-asan (PR): Python/Ruby/Node extensions built with
-fsanitize=address, run under their own suites.ports (PR): 32-bit (-m32 ILP32) and big-endian (s390x under
qemu-user) through scripts/check-ports.sh.api-docs (PR): Doxygen renders the public headers, uploaded as an
artifact.codegen-linux / codegen-apple (nightly): every binding's
generated-model test, split by toolchain family.make api-docs (Doxyfile): API reference from include/, the protobuf
importer headers included; output lands in build/doxygen/html.ROADMAP.md: what is done, what is never, what is next.docs/examples.md: one runnable round trip per binding, with the exact command.src/wkt.c (pbw_any_pack,
pbw_any_type_url/payload, pbw_timestamp/duration_to_millis) over field
numbers only; the core still special-cases no WKT.bindings/embedded/: static-memory facade over the C core for
microcontrollers — pbwe_open/build/decode/encode over one pbw_ctx,
caller-owned encode buffers, pbwe_set_limits for bounded input,
pbwe_open's descriptor limit for the startup allocation and
pbwe_decode_desc for a type fixed at build time. Covered by make test
and ctest, formatted and gated with the rest of the C sources..proto file or message named after a host keyword
produced code that did not compile. Go, Rust and Zig now sanitize the
derived package/module identifiers (struct.proto becomes package
struct_pbwire, module r#struct, alias struct_pbwire), Dart imports
sibling models under an _imp_ prefix (a dep message named Value was
ambiguous with the runtime), Lua requires siblings under _imp_<stem> (a dep
file named meta shadowed the file's own schema module and a dep named after
a Lua keyword was a syntax error), and a oneof enum member in the Java model
emitted Kind.KindCase instead of Kind<Prop>Case.src/wkt.c: pbw_timestamp/duration_to_millis() could overflow int64
when the nanosecond remainder pushed a seconds value at the edge of the
representable millisecond range over the bound (signed-integer overflow is
undefined behaviour). The range check now leaves room for the sub-second
part and reports PBW_ERR_LIMIT.src/wire.c: the non-'.' decimal-separator fallback could return an end
pointer before the start of the number when the host locale's separator is
multi-byte and the parse stopped inside it.src/json.c: the JSON printer recursed over nested messages without the
PBW_MAX_DEPTH cap the text printer and both codecs apply, so a hand-built
tree deep enough to pass pbw_message_new's checks could overflow the stack
in pbw_json_print(). It now reports PBW_ERR_LIMIT at the same depth as
every other formatter.pbwire_elixir never reached Hex in any release, because
mix hex.publish runs mix docs first and the docs task aborts when
ExDoc is not a dependency. ExDoc is a only: :dev, runtime: false
dependency now, so the published package carries HexDocs without costing
consumers a dependency. The nested else: inside Pbwire.Schema.nif_opts/1
is parenthesized (Elixir 1.20 warns about the keyword ambiguity).Schema.load() / Schema.from_file() leaked the partially built
schema pbw_descriptor_set_load() hands back on failure, and reported only
"invalid data" instead of the loader's reason (every other binding already
frees and surfaces it).LEN) run for a repeated closed (proto2)
enum whose field number is 1..15 was swallowed whole into the unknown bytes,
because the link-time one-byte tag map omitted the packed-form entry for
closed enums. The reference parser keeps known values in the field and moves
each unknown value into the unknown set as one unpacked item; the map now
routes that form to read_packed(), which does the same. (Multi-byte tags,
resolved through fnum_map, always took this path.) An all-unknown run now
leaves the field absent, like an empty packed run, instead of presenting an
empty list.Big-endian correctness, found by the new s390x gate:
encode.c: precomputed tag words were written with a 32-bit store, which
byte-reverses on a big-endian host (every tag was wrong). They now go
through w_tag_word_le, which keeps the one-store fast path where the host
byte order matches the wire and spells it byte by byte elsewhere.uint32_t first (same for the parser direction).bindings/c/runtime/pbw_struct.c: fixed-width member reads and writes
copied 4 bytes into/out of a uint64_t, which lands in the wrong half on
big-endian. ld_member/st_member go through the matching integer width.-m32) correctness: the struct self-check compared doubles with
==, which x87 excess precision makes unreliable; it compares bit patterns
now (double_bits).\\u surrogate-pair
combining and Any store-error normalization).sanitizers (ASan+UBSan make test), difftest-short
(fixed-seed differential run), fuzz-short, and coverage (core-only
gcov floor) jobs; conformance also runs on a nightly schedule.bindings/embedded: static-schema facade over the C core, including the
length-delimited encode/decode entry points.pbw_encode_delimited_to(),
pbw_decode_delimited_ctx[_limits](), and
pbw_descriptor_set_load_limits() for hostile descriptor sets.FeatureSet and MessageSet wire format
are loaded and queryable; the core still does no RPC.pbw_limits depth propagation through JSON and text parsing,
list form accepted in text format, unused internal helpers removed.Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |