The chunked nippy stream framing every serialization in the engine writes to a file or a sink — one home, so the export dump, the import reader and the snapshot sink cannot drift.
A chunked stream is a run of [int32 length][compressed chunk], each chunk an
independent compression window over back-to-back nippy frames. Constant memory on both
sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on
demand and drops it. A frame is whatever the caller freezes — a record's field map, an
index [key value] pair, a JTMS [handle label] pair — this namespace neither reads a
frame nor cares what one is.
Why it is its own namespace and not export's private helper. Three callers write
and read this format — vaelii.impl.io.export (the dump), vaelii.impl.io.import (the
reader) and vaelii.impl.io.snapshot (the derived-state image) — and the snapshot sink
is meant to be a thin adapter over this framing, not a second copy of it. A second
copy is exactly the drift a shared projection was created to avoid one layer up: two
writers that agree today and diverge on the next compression tweak. So the framing lives
once, here, and the container-specific concerns (a dump's meta.edn, an image's
manifest, which streams exist) stay with each caller.
The legacy single-window reader (read-window-seq) is kept for a v4/v5 foreign dump that
wrote one compression window over the whole file; the engine's own dumps have been chunked
since v6.
The chunked **nippy stream** framing every serialization in the engine writes to a file or a sink — one home, so the export dump, the import reader and the snapshot sink cannot drift. A chunked stream is a run of `[int32 length][compressed chunk]`, each chunk an independent compression window over back-to-back nippy frames. Constant memory on both sides: the writer holds one chunk and never the corpus, and the reader thaws a chunk on demand and drops it. A *frame* is whatever the caller freezes — a record's field map, an index `[key value]` pair, a JTMS `[handle label]` pair — this namespace neither reads a frame nor cares what one is. **Why it is its own namespace and not `export`'s private helper.** Three callers write and read this format — `vaelii.impl.io.export` (the dump), `vaelii.impl.io.import` (the reader) and `vaelii.impl.io.snapshot` (the derived-state image) — and the snapshot sink is meant to be *a thin adapter over this framing, not a second copy of it*. A second copy is exactly the drift a shared projection was created to avoid one layer up: two writers that agree today and diverge on the next compression tweak. So the framing lives once, here, and the container-specific concerns (a dump's `meta.edn`, an image's manifest, which streams exist) stay with each caller. The legacy single-window reader (`read-window-seq`) is kept for a v4/v5 foreign dump that wrote one compression window over the whole file; the engine's own dumps have been chunked since v6.
(read-chunked-seq file compression)Lazy seq of frames from a v6+ chunked stream file: a run of [int32 length] [compressed chunk]. Chunks are read serially and thawed on demand, so the whole
file never sits in heap. The stream closes when fully consumed.
Lazy seq of frames from a v6+ chunked stream file: a run of `[int32 length] [compressed chunk]`. Chunks are read serially and thawed on demand, so the whole file never sits in heap. The stream closes when fully consumed.
(read-window-seq file compression)Lazy seq of frames from a legacy v4/v5 stream file — one compression window over back-to-back frames, read serially. The stream closes when fully consumed.
Lazy seq of frames from a legacy v4/v5 stream file — one compression window over back-to-back frames, read serially. The stream closes when fully consumed.
(write-frames! file frames {:keys [compression chunk-size on-chunk]})Write frames into file in the chunked layout, chunk-size frames per chunk.
Returns how many frames were written.
Lazy by construction and by contract: one chunk is realized, frozen, compressed and
flushed before the next is asked for, so the writer's footprint is a chunk rather
than a corpus. :on-chunk is called with the running frame count at each chunk
boundary — the progress hook, and the point a caller's callback can throw to cancel.
Write `frames` into `file` in the chunked layout, `chunk-size` frames per chunk. Returns how many frames were written. Lazy by construction and by contract: one chunk is realized, frozen, compressed and flushed before the next is asked for, so the writer's footprint is a chunk rather than a corpus. `:on-chunk` is called with the running frame count at each chunk boundary — the progress hook, and the point a caller's callback can throw to cancel.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |