The read-only mount: a KvBackend and a RecordStore that answer every read and
refuse every write.
A base shared by N forks is only shared if nothing can write it, and the honest way to guarantee that is structurally rather than by review: an overlay composes over one of these, so a write path that forgot to divert fails loudly at the seam instead of silently mutating what every other fork is reading. That is invariant 1 of docs/overlay.md, held by construction.
Two calls are deliberately not refusals.
next-id on the frozen record store. It hands out a handle nobody holds and
touches no stored record, and it is what the overlay's id watermark is seeded from
(vaelii.impl.overlay.store) — the alternative, a max over the base's whole live-id
set, is O(base) at every mount. Two JVMs mounting one base each bump their own
in-RAM counter to the same value and then diverge into their own overlays, which is
exactly what independent forks are.kv-entries / sentex-ids and friends. Enumeration is a read.This is a decorator, not a file mode: it says nothing about how the underlying store was opened. Opening the base's files read-only at the OS level is the disk backend's business and orthogonal — this is what makes the composition safe whatever the base is (memory, disk, or a later SQL store).
The read-only mount: a `KvBackend` and a `RecordStore` that answer every read and **refuse every write**. A base shared by N forks is only shared if nothing can write it, and the honest way to guarantee that is structurally rather than by review: an overlay composes over one of these, so a write path that forgot to divert fails loudly at the seam instead of silently mutating what every other fork is reading. That is invariant 1 of docs/overlay.md, held by construction. Two calls are deliberately *not* refusals. * `next-id` on the frozen record store. It hands out a handle nobody holds and touches no stored record, and it is what the overlay's id watermark is seeded from (`vaelii.impl.overlay.store`) — the alternative, a `max` over the base's whole live-id set, is O(base) at every mount. Two JVMs mounting one base each bump their own in-RAM counter to the same value and then diverge into their own overlays, which is exactly what independent forks are. * `kv-entries` / `sentex-ids` and friends. Enumeration is a read. This is a decorator, not a file mode: it says nothing about how the underlying store was opened. Opening the base's files read-only at the OS level is the disk backend's business and orthogonal — this is what makes the *composition* safe whatever the base is (memory, disk, or a later SQL store).
OverlayKv — a composite KvBackend layering a private writable overlay over a
shared read-only base. Reads resolve overlay-first and fall through to the base;
writes land only in the overlay; the base is never mutated, so N JVMs can share one
frozen base index while each keeps its own fork.
This is the index half of the :overlay backend (the record half is
vaelii.impl.overlay.store), and it is the whole of it. KvIndexStore
(vaelii.impl.kv) writes every index family — the count-aware trie, the context /
functor / argument roots, the rule index, the exception re-check index, the inverted
term index and the term roster — in terms of this protocol and holds no state of its
own, so one decorator here forks the entire index and the trie walker, the matcher, the
planner and the query layers above it are unchanged. That is what the KvBackend seam
was extracted for.
members(K) = (base(K) ∪ overlay(K)) − removed(K), where base(K) is
empty if K carries a tombstone, and removed(K) records the base members this
overlay removed (a base set cannot be edited, so a removal is recorded rather than
applied).kv-increment / kv-decrement seeds the overlay from
the base value, so afterwards the overlay holds base+net and reads are exact. An
untouched counter reads straight through.::deleted-keys shadows the base for
that key. A later kv-add-to-set repopulates it from the overlay only — the base
stays shadowed, which is "deleted, then re-added fresh". kv-put shadows the same
way, because a put replaces a key rather than merging into it.kv-clear! empties the overlay and sets ::cleared, after
which every base key reads absent. That is O(1) rather than a tombstone per base
key, and it is what makes reindex work on a fork: clear the merged index, then
rebuild it from the merged records.Bookkeeping lives in the overlay under reserved keys — ::cleared, ::deleted-keys,
and [::removed K] — namespaced here, so they cannot collide with an index key (every
one of those is a vector tagged with an unnamespaced keyword, or [:term-roster]).
Because the bookkeeping is overlay data, a durable overlay carries it durably and a
remount serves the same merged view with no separate recovery step.
kv-count answers the merged cardinality, never the overlay's. The count-aware
trie is a selectivity structure — plan/order costs every conjunct off count-at, and
provers/est-bindings off the functor root — so a base-blind count would not be a
wrong answer, it would be a silently wrong plan for every query touching inherited
content. kv-intersect merges for the same reason: sentexes-with-args is one set
intersection over the functor and argument roots, and it must see the base's postings
or a fork would stop finding its own inherited facts.
Merging is not the same as building the merged set, and the difference is the cost of
every query plan a fork makes. A key the fork has not touched — no overlay members, no
recorded removal, no tombstone — merges to the base's own value, so inherited? names
that case and kv-count / kv-members hand the base's own answer straight back. Over
a flat-map base the two roads read the same, because its kv-members is a reference
return; over a :dense base, where the posting has to be materialized into a set first,
counting through the merge cost 12.6 ms per call on a 100,000-handle root — a selectivity
read, per conjunct, on a key the fork had never written to. kv-member? is the same
observation at member granularity: exception-rule? probes both sides rather than
merging, which is what keeps the firing-path gate O(1) across the seam.
kv-get is the scalar/counter read and does not merge set values: an overlay value
shadows the base's. No key in the index is read both ways — the trie's counters are
read by kv-get and its handle sets by kv-members — and a backend is free to hold a
posting in a private representation (vaelii.impl.dense-kv returns an IntPostings
here), so merging at this op would mean type-testing another backend's internals.
kv-entries, whose contract is Clojure sets, merges properly.
Single writer. Pure runs one (docs/storage.md). A batch is applied op by op
through this decorator rather than as one atomic step the way MemoryKvBackend applies
it, so the instance lock is held around the whole of kv-batch — an incidental reader
beside the writer then sees a batch whole or not at all, as it does on every other
backend.
`OverlayKv` — a composite `KvBackend` layering a private **writable** overlay over a shared **read-only** base. Reads resolve overlay-first and fall through to the base; writes land only in the overlay; the base is never mutated, so N JVMs can share one frozen base index while each keeps its own fork. This is the index half of the `:overlay` backend (the record half is `vaelii.impl.overlay.store`), and it is the whole of it. `KvIndexStore` (`vaelii.impl.kv`) writes *every* index family — the count-aware trie, the context / functor / argument roots, the rule index, the exception re-check index, the inverted term index and the term roster — in terms of this protocol and holds no state of its own, so one decorator here forks the entire index and the trie walker, the matcher, the planner and the query layers above it are unchanged. That is what the `KvBackend` seam was extracted for. ## The merge model, per key * **Sets.** `members(K) = (base(K) ∪ overlay(K)) − removed(K)`, where `base(K)` is empty if `K` carries a tombstone, and `removed(K)` records the base members this overlay removed (a base set cannot be edited, so a removal is recorded rather than applied). * **Scalars and counters.** An overlay value shadows the base's. A counter is **copy-on-write**: the first `kv-increment` / `kv-decrement` seeds the overlay from the base value, so afterwards the overlay holds base+net and reads are exact. An untouched counter reads straight through. * **Whole-key delete.** A sticky tombstone in `::deleted-keys` shadows the base for that key. A later `kv-add-to-set` repopulates it from the overlay *only* — the base stays shadowed, which is "deleted, then re-added fresh". `kv-put` shadows the same way, because a put replaces a key rather than merging into it. * **Wholesale clear.** `kv-clear!` empties the overlay and sets `::cleared`, after which every base key reads absent. That is O(1) rather than a tombstone per base key, and it is what makes `reindex` work on a fork: clear the merged index, then rebuild it from the merged records. Bookkeeping lives in the overlay under reserved keys — `::cleared`, `::deleted-keys`, and `[::removed K]` — namespaced here, so they cannot collide with an index key (every one of those is a vector tagged with an unnamespaced keyword, or `[:term-roster]`). Because the bookkeeping *is* overlay data, a durable overlay carries it durably and a remount serves the same merged view with no separate recovery step. ## What has to be merged, and why `kv-count` answers the **merged** cardinality, never the overlay's. The count-aware trie is a selectivity structure — `plan/order` costs every conjunct off `count-at`, and `provers/est-bindings` off the functor root — so a base-blind count would not be a wrong answer, it would be a silently wrong *plan* for every query touching inherited content. `kv-intersect` merges for the same reason: `sentexes-with-args` is one set intersection over the functor and argument roots, and it must see the base's postings or a fork would stop finding its own inherited facts. Merging is not the same as *building* the merged set, and the difference is the cost of every query plan a fork makes. A key the fork has not touched — no overlay members, no recorded removal, no tombstone — merges to the base's own value, so `inherited?` names that case and `kv-count` / `kv-members` hand the base's own answer straight back. Over a flat-map base the two roads read the same, because its `kv-members` is a reference return; over a `:dense` base, where the posting has to be materialized into a set first, counting through the merge cost 12.6 ms per call on a 100,000-handle root — a selectivity read, per conjunct, on a key the fork had never written to. `kv-member?` is the same observation at member granularity: `exception-rule?` probes both sides rather than merging, which is what keeps the firing-path gate O(1) across the seam. `kv-get` is the scalar/counter read and does **not** merge set values: an overlay value shadows the base's. No key in the index is read both ways — the trie's counters are read by `kv-get` and its handle sets by `kv-members` — and a backend is free to hold a posting in a private representation (`vaelii.impl.dense-kv` returns an `IntPostings` here), so merging at this op would mean type-testing another backend's internals. `kv-entries`, whose contract *is* Clojure sets, merges properly. **Single writer.** Pure runs one (docs/storage.md). A batch is applied op by op through this decorator rather than as one atomic step the way `MemoryKvBackend` applies it, so the instance lock is held around the whole of `kv-batch` — an incidental reader beside the writer then sees a batch whole or not at all, as it does on every other backend.
Mounting a fork: freeze a base, compose a private writable overlay over it, and hand back the two stores a KB is built from.
A base is any pair of stores mounted read-only (vaelii.impl.overlay.frozen); a
fork is a fresh writable pair composed over it. Nothing is copied and nothing in
the base is written, so N JVMs mount one frozen base and each evolves its own fork —
the sharing needs no protocol between them, and equally offers no coherence between
them: a base that changes under a mounted fork is outside the contract.
The overlay is a KvBackend decorator, so it forks exactly the index path that is
written over that protocol: KvIndexStore (vaelii.impl.kv) and therefore the
:memory, :dense and :disk index axes. The :columnar index is a native
IndexStore — its trie is int-id nodes in parallel arrays, with no keys and no backend
underneath — so a KvBackend decorator would fork its roots and leave its trie behind.
That is refused here rather than half-done. Forking a columnar index is a different
construction: its compacted CSR mode is already an immutable base, so the natural shape
is a mutable columnar head over a frozen CSR base, not a KV decorator.
The record half keeps tombstones and released premise marks in a small KvBackend
beside the overlay (vaelii.impl.overlay.store). An in-RAM overlay gets an in-RAM one
— the whole fork is ephemeral — and a disk overlay gets a durable one under
<dir>/overlay-meta, so remounting that directory over the same base serves the merged
view it was left in. The index half needs none: its bookkeeping lives in the overlay
index itself, under reserved keys, and so is exactly as durable as the fork is.
Mounting a fork: freeze a base, compose a private writable overlay over it, and hand back the two stores a KB is built from. A **base** is any pair of stores mounted read-only (`vaelii.impl.overlay.frozen`); a **fork** is a fresh writable pair composed over it. Nothing is copied and nothing in the base is written, so N JVMs mount one frozen base and each evolves its own fork — the sharing needs no protocol between them, and equally offers no coherence between them: a base that changes under a mounted fork is outside the contract. ## Which index a fork can be taken over The overlay is a `KvBackend` decorator, so it forks exactly the index path that is written over that protocol: `KvIndexStore` (`vaelii.impl.kv`) and therefore the `:memory`, `:dense` and `:disk` index axes. The `:columnar` index is a **native** `IndexStore` — its trie is int-id nodes in parallel arrays, with no keys and no backend underneath — so a `KvBackend` decorator would fork its roots and leave its trie behind. That is refused here rather than half-done. Forking a columnar index is a different construction: its compacted CSR mode is already an immutable base, so the natural shape is a mutable columnar head over a frozen CSR base, not a KV decorator. ## Bookkeeping The record half keeps tombstones and released premise marks in a small `KvBackend` beside the overlay (`vaelii.impl.overlay.store`). An in-RAM overlay gets an in-RAM one — the whole fork is ephemeral — and a disk overlay gets a durable one under `<dir>/overlay-meta`, so remounting that directory over the same base serves the merged view it was left in. The index half needs none: its bookkeeping lives in the overlay index itself, under reserved keys, and so is exactly as durable as the fork is.
OverlayRecordStore — a composite RecordStore layering a private writable
overlay over a shared read-only base. Reads resolve overlay-first, skipping
tombstoned base handles; writes land only in the overlay; the base is never mutated.
The record half of the :overlay backend (the index half is
vaelii.impl.overlay.kv).
mark-premise materializes an override
before it writes, since the assumption strength lives on the record.clear-records! empties the overlay and marks the base
hidden — one flag rather than a tombstone per base handle — so a fork can be reset
to empty without walking what it inherited.KvBackend under reserved keys, mirrored in atoms for
the read path. Every mutation writes through, and a mount rebuilds the atoms from
it, so remounting a durable overlay over the same base serves the same merged view:
a deleted base record stays deleted, a released premise stays released. An
ephemeral fork passes an in-RAM bookkeeping backend and pays nothing for the
machinery.Counts need no delta bookkeeping here. A RecordStore in pure exposes handle
sets rather than counts, and everything counted — sentex-count, context-size,
count-with-functor — is read off the index, where the merge is the trie's own
copy-on-write counters and the merged root sets (vaelii.impl.overlay.kv). So the
counts a fork reports are exact by construction rather than by a second, parallel
accounting that could drift from the records.
The fork's belief is rebuilt, not overlaid. The JTMS is not storage — it is a
separate protocol (vaelii.impl.jtms) over derived state — and pure already has the
operation that computes it from records: recover. So a fork gets its own network by
recovering over the merged view, and nothing here layers one truth-maintenance graph
over another.
`OverlayRecordStore` — a composite `RecordStore` layering a private **writable** overlay over a shared **read-only** base. Reads resolve overlay-first, skipping tombstoned base handles; writes land only in the overlay; the base is never mutated. The record half of the `:overlay` backend (the index half is `vaelii.impl.overlay.kv`). * **The id seam.** The overlay's handle counter is seeded above every handle the base holds, so a newly minted handle can never collide with a base one. A record written at a handle the base *already* uses is therefore an **override** — the same handle, a different record — and the overlay's copy wins every read. That is how a base record is edited without editing the base: `mark-premise` materializes an override before it writes, since the assumption strength lives on the record. * **Tombstones.** Deleting a base handle cannot touch the base, so it is recorded and the read path filters it. They are sticky: a base record cannot come back through fall-through, only by being written again into the overlay (a revival, at the same handle). * **Wholesale clear.** `clear-records!` empties the overlay and marks the base hidden — one flag rather than a tombstone per base handle — so a fork can be reset to empty without walking what it inherited. * **Durable bookkeeping.** The tombstone sets, the released premise marks and the hidden flag live in a small `KvBackend` under reserved keys, mirrored in atoms for the read path. Every mutation writes through, and a mount rebuilds the atoms from it, so remounting a durable overlay over the same base serves the same merged view: a deleted base record stays deleted, a released premise stays released. An ephemeral fork passes an in-RAM bookkeeping backend and pays nothing for the machinery. **Counts need no delta bookkeeping here.** A `RecordStore` in pure exposes handle *sets* rather than counts, and everything counted — `sentex-count`, `context-size`, `count-with-functor` — is read off the index, where the merge is the trie's own copy-on-write counters and the merged root sets (`vaelii.impl.overlay.kv`). So the counts a fork reports are exact by construction rather than by a second, parallel accounting that could drift from the records. **The fork's belief is rebuilt, not overlaid.** The JTMS is not storage — it is a separate protocol (`vaelii.impl.jtms`) over derived state — and pure already has the operation that computes it from records: `recover`. So a fork gets its own network by recovering over the merged view, and nothing here layers one truth-maintenance graph over another.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |