Liking cljdoc? Tell your friends :D

vaelii.impl.overlay.frozen

The read-only mount: a KvBackend and a RecordStore that answer every read and refuse every write.

A base shared by N forks is only shared if nothing can write it, and the honest way to guarantee that is structurally rather than by review: an overlay composes over one of these, so a write path that forgot to divert fails loudly at the seam instead of silently mutating what every other fork is reading. That is invariant 1 of docs/overlay.md, held by construction.

Two calls are deliberately not refusals.

  • next-id on the frozen record store. It hands out a handle nobody holds and touches no stored record, and it is what the overlay's id watermark is seeded from (vaelii.impl.overlay.store) — the alternative, a max over the base's whole live-id set, is O(base) at every mount. Two JVMs mounting one base each bump their own in-RAM counter to the same value and then diverge into their own overlays, which is exactly what independent forks are.
  • kv-entries / sentex-ids and friends. Enumeration is a read.

This is a decorator, not a file mode: it says nothing about how the underlying store was opened. Opening the base's files read-only at the OS level is the disk backend's business and orthogonal — this is what makes the composition safe whatever the base is (memory, disk, or a later SQL store).

The read-only mount: a `KvBackend` and a `RecordStore` that answer every read and
**refuse every write**.

A base shared by N forks is only shared if nothing can write it, and the honest way to
guarantee that is structurally rather than by review: an overlay composes over one of
these, so a write path that forgot to divert fails loudly at the seam instead of
silently mutating what every other fork is reading.  That is invariant 1 of
docs/overlay.md, held by construction.

Two calls are deliberately *not* refusals.

* `next-id` on the frozen record store.  It hands out a handle nobody holds and
  touches no stored record, and it is what the overlay's id watermark is seeded from
  (`vaelii.impl.overlay.store`) — the alternative, a `max` over the base's whole live-id
  set, is O(base) at every mount.  Two JVMs mounting one base each bump their own
  in-RAM counter to the same value and then diverge into their own overlays, which is
  exactly what independent forks are.
* `kv-entries` / `sentex-ids` and friends.  Enumeration is a read.

This is a decorator, not a file mode: it says nothing about how the underlying store
was opened.  Opening the base's files read-only at the OS level is the disk backend's
business and orthogonal — this is what makes the *composition* safe whatever the base
is (memory, disk, or a later SQL store).
raw docstring

vaelii.impl.overlay.kv

OverlayKv — a composite KvBackend layering a private writable overlay over a shared read-only base. Reads resolve overlay-first and fall through to the base; writes land only in the overlay; the base is never mutated, so N JVMs can share one frozen base index while each keeps its own fork.

This is the index half of the :overlay backend (the record half is vaelii.impl.overlay.store), and it is the whole of it. KvIndexStore (vaelii.impl.kv) writes every index family — the count-aware trie, the context / functor / argument roots, the rule index, the exception re-check index, the inverted term index and the term roster — in terms of this protocol and holds no state of its own, so one decorator here forks the entire index and the trie walker, the matcher, the planner and the query layers above it are unchanged. That is what the KvBackend seam was extracted for.

The merge model, per key

  • Sets. members(K) = (base(K) ∪ overlay(K)) − removed(K), where base(K) is empty if K carries a tombstone, and removed(K) records the base members this overlay removed (a base set cannot be edited, so a removal is recorded rather than applied).
  • Scalars and counters. An overlay value shadows the base's. A counter is copy-on-write: the first kv-increment / kv-decrement seeds the overlay from the base value, so afterwards the overlay holds base+net and reads are exact. An untouched counter reads straight through.
  • Whole-key delete. A sticky tombstone in ::deleted-keys shadows the base for that key. A later kv-add-to-set repopulates it from the overlay only — the base stays shadowed, which is "deleted, then re-added fresh". kv-put shadows the same way, because a put replaces a key rather than merging into it.
  • Wholesale clear. kv-clear! empties the overlay and sets ::cleared, after which every base key reads absent. That is O(1) rather than a tombstone per base key, and it is what makes reindex work on a fork: clear the merged index, then rebuild it from the merged records.

Bookkeeping lives in the overlay under reserved keys — ::cleared, ::deleted-keys, and [::removed K] — namespaced here, so they cannot collide with an index key (every one of those is a vector tagged with an unnamespaced keyword, or [:term-roster]). Because the bookkeeping is overlay data, a durable overlay carries it durably and a remount serves the same merged view with no separate recovery step.

What has to be merged, and why

kv-count answers the merged cardinality, never the overlay's. The count-aware trie is a selectivity structure — plan/order costs every conjunct off count-at, and provers/est-bindings off the functor root — so a base-blind count would not be a wrong answer, it would be a silently wrong plan for every query touching inherited content. kv-intersect merges for the same reason: sentexes-with-args is one set intersection over the functor and argument roots, and it must see the base's postings or a fork would stop finding its own inherited facts.

Merging is not the same as building the merged set, and the difference is the cost of every query plan a fork makes. A key the fork has not touched — no overlay members, no recorded removal, no tombstone — merges to the base's own value, so inherited? names that case and kv-count / kv-members hand the base's own answer straight back. Over a flat-map base the two roads read the same, because its kv-members is a reference return; over a :dense base, where the posting has to be materialized into a set first, counting through the merge cost 12.6 ms per call on a 100,000-handle root — a selectivity read, per conjunct, on a key the fork had never written to. kv-member? is the same observation at member granularity: exception-rule? probes both sides rather than merging, which is what keeps the firing-path gate O(1) across the seam.

kv-get is the scalar/counter read and does not merge set values: an overlay value shadows the base's. No key in the index is read both ways — the trie's counters are read by kv-get and its handle sets by kv-members — and a backend is free to hold a posting in a private representation (vaelii.impl.dense-kv returns an IntPostings here), so merging at this op would mean type-testing another backend's internals. kv-entries, whose contract is Clojure sets, merges properly.

Single writer. Pure runs one (docs/storage.md). A batch is applied op by op through this decorator rather than as one atomic step the way MemoryKvBackend applies it, so the instance lock is held around the whole of kv-batch — an incidental reader beside the writer then sees a batch whole or not at all, as it does on every other backend.

`OverlayKv` — a composite `KvBackend` layering a private **writable** overlay over a
shared **read-only** base.  Reads resolve overlay-first and fall through to the base;
writes land only in the overlay; the base is never mutated, so N JVMs can share one
frozen base index while each keeps its own fork.

This is the index half of the `:overlay` backend (the record half is
`vaelii.impl.overlay.store`), and it is the whole of it.  `KvIndexStore`
(`vaelii.impl.kv`) writes *every* index family — the count-aware trie, the context /
functor / argument roots, the rule index, the exception re-check index, the inverted
term index and the term roster — in terms of this protocol and holds no state of its
own, so one decorator here forks the entire index and the trie walker, the matcher, the
planner and the query layers above it are unchanged.  That is what the `KvBackend` seam
was extracted for.

## The merge model, per key

* **Sets.**  `members(K) = (base(K) ∪ overlay(K)) − removed(K)`, where `base(K)` is
  empty if `K` carries a tombstone, and `removed(K)` records the base members this
  overlay removed (a base set cannot be edited, so a removal is recorded rather than
  applied).
* **Scalars and counters.**  An overlay value shadows the base's.  A counter is
  **copy-on-write**: the first `kv-increment` / `kv-decrement` seeds the overlay from
  the base value, so afterwards the overlay holds base+net and reads are exact.  An
  untouched counter reads straight through.
* **Whole-key delete.**  A sticky tombstone in `::deleted-keys` shadows the base for
  that key.  A later `kv-add-to-set` repopulates it from the overlay *only* — the base
  stays shadowed, which is "deleted, then re-added fresh".  `kv-put` shadows the same
  way, because a put replaces a key rather than merging into it.
* **Wholesale clear.**  `kv-clear!` empties the overlay and sets `::cleared`, after
  which every base key reads absent.  That is O(1) rather than a tombstone per base
  key, and it is what makes `reindex` work on a fork: clear the merged index, then
  rebuild it from the merged records.

Bookkeeping lives in the overlay under reserved keys — `::cleared`, `::deleted-keys`,
and `[::removed K]` — namespaced here, so they cannot collide with an index key (every
one of those is a vector tagged with an unnamespaced keyword, or `[:term-roster]`).
Because the bookkeeping *is* overlay data, a durable overlay carries it durably and a
remount serves the same merged view with no separate recovery step.

## What has to be merged, and why

`kv-count` answers the **merged** cardinality, never the overlay's.  The count-aware
trie is a selectivity structure — `plan/order` costs every conjunct off `count-at`, and
`provers/est-bindings` off the functor root — so a base-blind count would not be a
wrong answer, it would be a silently wrong *plan* for every query touching inherited
content.  `kv-intersect` merges for the same reason: `sentexes-with-args` is one set
intersection over the functor and argument roots, and it must see the base's postings
or a fork would stop finding its own inherited facts.

Merging is not the same as *building* the merged set, and the difference is the cost of
every query plan a fork makes.  A key the fork has not touched — no overlay members, no
recorded removal, no tombstone — merges to the base's own value, so `inherited?` names
that case and `kv-count` / `kv-members` hand the base's own answer straight back.  Over
a flat-map base the two roads read the same, because its `kv-members` is a reference
return; over a `:dense` base, where the posting has to be materialized into a set first,
counting through the merge cost 12.6 ms per call on a 100,000-handle root — a selectivity
read, per conjunct, on a key the fork had never written to.  `kv-member?` is the same
observation at member granularity: `exception-rule?` probes both sides rather than
merging, which is what keeps the firing-path gate O(1) across the seam.

`kv-get` is the scalar/counter read and does **not** merge set values: an overlay value
shadows the base's.  No key in the index is read both ways — the trie's counters are
read by `kv-get` and its handle sets by `kv-members` — and a backend is free to hold a
posting in a private representation (`vaelii.impl.dense-kv` returns an `IntPostings`
here), so merging at this op would mean type-testing another backend's internals.
`kv-entries`, whose contract *is* Clojure sets, merges properly.

**Single writer.**  Pure runs one (docs/storage.md).  A batch is applied op by op
through this decorator rather than as one atomic step the way `MemoryKvBackend` applies
it, so the instance lock is held around the whole of `kv-batch` — an incidental reader
beside the writer then sees a batch whole or not at all, as it does on every other
backend.
raw docstring

vaelii.impl.overlay.mount

Mounting a fork: freeze a base, compose a private writable overlay over it, and hand back the two stores a KB is built from.

A base is any pair of stores mounted read-only (vaelii.impl.overlay.frozen); a fork is a fresh writable pair composed over it. Nothing is copied and nothing in the base is written, so N JVMs mount one frozen base and each evolves its own fork — the sharing needs no protocol between them, and equally offers no coherence between them: a base that changes under a mounted fork is outside the contract.

Which index a fork can be taken over

The overlay is a KvBackend decorator, so it forks exactly the index path that is written over that protocol: KvIndexStore (vaelii.impl.kv) and therefore the :memory, :dense and :disk index axes. The :columnar index is a native IndexStore — its trie is int-id nodes in parallel arrays, with no keys and no backend underneath — so a KvBackend decorator would fork its roots and leave its trie behind. That is refused here rather than half-done. Forking a columnar index is a different construction: its compacted CSR mode is already an immutable base, so the natural shape is a mutable columnar head over a frozen CSR base, not a KV decorator.

Bookkeeping

The record half keeps tombstones and released premise marks in a small KvBackend beside the overlay (vaelii.impl.overlay.store). An in-RAM overlay gets an in-RAM one — the whole fork is ephemeral — and a disk overlay gets a durable one under <dir>/overlay-meta, so remounting that directory over the same base serves the merged view it was left in. The index half needs none: its bookkeeping lives in the overlay index itself, under reserved keys, and so is exactly as durable as the fork is.

Mounting a fork: freeze a base, compose a private writable overlay over it, and hand
back the two stores a KB is built from.

A **base** is any pair of stores mounted read-only (`vaelii.impl.overlay.frozen`); a
**fork** is a fresh writable pair composed over it.  Nothing is copied and nothing in
the base is written, so N JVMs mount one frozen base and each evolves its own fork —
the sharing needs no protocol between them, and equally offers no coherence between
them: a base that changes under a mounted fork is outside the contract.

## Which index a fork can be taken over

The overlay is a `KvBackend` decorator, so it forks exactly the index path that is
written over that protocol: `KvIndexStore` (`vaelii.impl.kv`) and therefore the
`:memory`, `:dense` and `:disk` index axes.  The `:columnar` index is a **native**
`IndexStore` — its trie is int-id nodes in parallel arrays, with no keys and no backend
underneath — so a `KvBackend` decorator would fork its roots and leave its trie behind.
That is refused here rather than half-done.  Forking a columnar index is a different
construction: its compacted CSR mode is already an immutable base, so the natural shape
is a mutable columnar head over a frozen CSR base, not a KV decorator.

## Bookkeeping

The record half keeps tombstones and released premise marks in a small `KvBackend`
beside the overlay (`vaelii.impl.overlay.store`).  An in-RAM overlay gets an in-RAM one
— the whole fork is ephemeral — and a disk overlay gets a durable one under
`<dir>/overlay-meta`, so remounting that directory over the same base serves the merged
view it was left in.  The index half needs none: its bookkeeping lives in the overlay
index itself, under reserved keys, and so is exactly as durable as the fork is.
raw docstring

vaelii.impl.overlay.store

OverlayRecordStore — a composite RecordStore layering a private writable overlay over a shared read-only base. Reads resolve overlay-first, skipping tombstoned base handles; writes land only in the overlay; the base is never mutated. The record half of the :overlay backend (the index half is vaelii.impl.overlay.kv).

  • The id seam. The overlay's handle counter is seeded above every handle the base holds, so a newly minted handle can never collide with a base one. A record written at a handle the base already uses is therefore an override — the same handle, a different record — and the overlay's copy wins every read. That is how a base record is edited without editing the base: mark-premise materializes an override before it writes, since the assumption strength lives on the record.
  • Tombstones. Deleting a base handle cannot touch the base, so it is recorded and the read path filters it. They are sticky: a base record cannot come back through fall-through, only by being written again into the overlay (a revival, at the same handle).
  • Wholesale clear. clear-records! empties the overlay and marks the base hidden — one flag rather than a tombstone per base handle — so a fork can be reset to empty without walking what it inherited.
  • Durable bookkeeping. The tombstone sets, the released premise marks and the hidden flag live in a small KvBackend under reserved keys, mirrored in atoms for the read path. Every mutation writes through, and a mount rebuilds the atoms from it, so remounting a durable overlay over the same base serves the same merged view: a deleted base record stays deleted, a released premise stays released. An ephemeral fork passes an in-RAM bookkeeping backend and pays nothing for the machinery.

Counts need no delta bookkeeping here. A RecordStore in pure exposes handle sets rather than counts, and everything counted — sentex-count, context-size, count-with-functor — is read off the index, where the merge is the trie's own copy-on-write counters and the merged root sets (vaelii.impl.overlay.kv). So the counts a fork reports are exact by construction rather than by a second, parallel accounting that could drift from the records.

The fork's belief is rebuilt, not overlaid. The JTMS is not storage — it is a separate protocol (vaelii.impl.jtms) over derived state — and pure already has the operation that computes it from records: recover. So a fork gets its own network by recovering over the merged view, and nothing here layers one truth-maintenance graph over another.

`OverlayRecordStore` — a composite `RecordStore` layering a private **writable**
overlay over a shared **read-only** base.  Reads resolve overlay-first, skipping
tombstoned base handles; writes land only in the overlay; the base is never mutated.
The record half of the `:overlay` backend (the index half is
`vaelii.impl.overlay.kv`).

* **The id seam.**  The overlay's handle counter is seeded above every handle the base
  holds, so a newly minted handle can never collide with a base one.  A record written
  at a handle the base *already* uses is therefore an **override** — the same handle,
  a different record — and the overlay's copy wins every read.  That is how a base
  record is edited without editing the base: `mark-premise` materializes an override
  before it writes, since the assumption strength lives on the record.
* **Tombstones.**  Deleting a base handle cannot touch the base, so it is recorded and
  the read path filters it.  They are sticky: a base record cannot come back through
  fall-through, only by being written again into the overlay (a revival, at the same
  handle).
* **Wholesale clear.**  `clear-records!` empties the overlay and marks the base
  hidden — one flag rather than a tombstone per base handle — so a fork can be reset
  to empty without walking what it inherited.
* **Durable bookkeeping.**  The tombstone sets, the released premise marks and the
  hidden flag live in a small `KvBackend` under reserved keys, mirrored in atoms for
  the read path.  Every mutation writes through, and a mount rebuilds the atoms from
  it, so remounting a durable overlay over the same base serves the same merged view:
  a deleted base record stays deleted, a released premise stays released.  An
  ephemeral fork passes an in-RAM bookkeeping backend and pays nothing for the
  machinery.

**Counts need no delta bookkeeping here.**  A `RecordStore` in pure exposes handle
*sets* rather than counts, and everything counted — `sentex-count`, `context-size`,
`count-with-functor` — is read off the index, where the merge is the trie's own
copy-on-write counters and the merged root sets (`vaelii.impl.overlay.kv`).  So the
counts a fork reports are exact by construction rather than by a second, parallel
accounting that could drift from the records.

**The fork's belief is rebuilt, not overlaid.**  The JTMS is not storage — it is a
separate protocol (`vaelii.impl.jtms`) over derived state — and pure already has the
operation that computes it from records: `recover`.  So a fork gets its own network by
recovering over the merged view, and nothing here layers one truth-maintenance graph
over another.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
Move to previous article
Move to next article
Ctrl+/Jump to the search field
× close