The durable-store entry point: open (once per directory) a DiskRecordStore and/or a
KvIndexStore over a DiskKvBackend, take the single-writer lock, and wire what was
opened to the durability daemon.
The two halves open independently. A KB's records and its index are chosen on
separate axes (vaelii.impl.kb), and only one of the combinations that reach here
wants both: :disk-log is durable records and a durable index, while :disk-memory /
:disk-dense / :disk-columnar keep the derived index in RAM and want the record store
alone — no index log, no index WAL, nothing written to the directory but the records. So each
component is opened on first use rather than as a pair, and the registry records which
ones a directory actually has. A directory opened both ways in one JVM therefore
shares its record store across both KBs and hands the RAM-index one no durable index
at all.
A process-global registry keyed by canonical directory mirrors the memory backend's space registry: two KBs constructed over the same directory share one set of stores — so a KB "restarted" over the same directory in one JVM (the recovery tests) sees the durable records the first wrote, with its own fresh taxonomy/TMS, and the file handles + durability registration + lock are taken once rather than leaking across the suite's hundreds of KB constructions. A true cross-JVM restart opens the directory fresh and rebuilds the RAM state from the durable logs.
The durable-store entry point: open (once per directory) a `DiskRecordStore` and/or a `KvIndexStore` over a `DiskKvBackend`, take the single-writer lock, and wire what was opened to the durability daemon. **The two halves open independently.** A KB's records and its index are chosen on separate axes (`vaelii.impl.kb`), and only one of the combinations that reach here wants both: `:disk-log` is durable records *and* a durable index, while `:disk-memory` / `:disk-dense` / `:disk-columnar` keep the derived index in RAM and want the record store alone — no index log, no index WAL, nothing written to the directory but the records. So each component is opened on first use rather than as a pair, and the registry records which ones a directory actually has. A directory opened both ways in one JVM therefore shares its record store across both KBs and hands the RAM-index one no durable index at all. A process-global registry keyed by canonical directory mirrors the memory backend's space registry: two KBs constructed over the same directory share one set of stores — so a KB "restarted" over the same directory in one JVM (the recovery tests) sees the durable records the first wrote, with its own fresh taxonomy/TMS, and the file handles + durability registration + lock are taken once rather than leaking across the suite's hundreds of KB constructions. A true cross-JVM restart opens the directory fresh and rebuilds the RAM state from the durable logs.
(canonical-dir dir)dir's canonical path — the identity a directory's stores are keyed by, and what
anything else keying state off a disk KB's directory (the derived index's shared
state, vaelii.impl.kb) must key on too, so two spellings of one directory are one
KB rather than two.
`dir`'s canonical path — the identity a directory's stores are keyed by, and what anything else keying state off a disk KB's directory (the derived index's shared state, `vaelii.impl.kb`) must key on too, so two spellings of one directory are one KB rather than two.
(close-dir! dir)Close and forget the stores for dir: fsync + close whichever components are open,
deregister them from the durability daemon, and release the lock. For explicit
shutdown and test teardown — ordinary operation leaves the stores open for the JVM's
life (the shutdown hook closes them).
This is what vaelii.core/close! exists for — handing the directory to another
process without exiting the JVM — so an unclean close must not defeat it: every
component gets its close attempt, the lock release and the registry removal run even
when one throws, and the first component failure is rethrown after that cleanup.
The caller learns the close did not complete cleanly, and the directory is still
released either way.
Close and forget the stores for `dir`: fsync + close whichever components are open, deregister them from the durability daemon, and release the lock. For explicit shutdown and test teardown — ordinary operation leaves the stores open for the JVM's life (the shutdown hook closes them). This is what `vaelii.core/close!` exists for — handing the directory to another process without exiting the JVM — so an unclean close must not defeat it: every component gets its close attempt, the lock release and the registry removal run even when one throws, and the first component failure is rethrown *after* that cleanup. The caller learns the close did not complete cleanly, and the directory is still released either way.
(disk-dir {:keys [dir space] :or {space 0}})The directory a disk KB lives in. :dir names it explicitly; otherwise it derives
from the space number under a base (vaelii.disk.dir, else <tmpdir>/vaelii-disk), so
the space the other backends key on still names a distinct store.
The directory a disk KB lives in. `:dir` names it explicitly; otherwise it derives from the space number under a base (`vaelii.disk.dir`, else `<tmpdir>/vaelii-disk`), so the space the other backends key on still names a distinct store.
(index-for dir)The :disk-log IndexStore for dir — the KvIndexStore over a write-ahead-logged
DiskKvBackend, whose map is in RAM and whose log buys the restart
(vaelii.impl.disk.kv).
The `:disk-log` `IndexStore` for `dir` — the `KvIndexStore` over a write-ahead-logged `DiskKvBackend`, whose map is in RAM and whose log buys the restart (`vaelii.impl.disk.kv`).
(maybe-refresh-index-snapshot! cdir roots-now)Rewrite cdir's index image if the live index has drifted past the threshold.
cdir is already canonical — the KB carries the resolved path (kb's
:snapshot-dir), so nothing here calls getCanonicalPath and the write door pays no
realpath per assert. A caller holding a directory as its user spelled it canonicalizes
it once and keeps the answer; this is the wrong place to do it per call.
Called on the writer's thread, from the write door (kb/create-sentex), because
that is the only thread allowed to touch the columnar index (disk/index_snapshot.clj,
"the cadence"). So the gate is asked per write, and what it costs is worth stating
rather than guessing at: one read of the registry here, which is where a directory with
no image writer registered stops — and then, for one that has, due?'s two property
reads, the canonical path its state is keyed by, one map read and a clock. The caller
reaches this at all only when its KB carries a :snapshot-dir, which is what keeps the
whole of it off every other backend's write path.
A refresh that throws must not fail the write. By the time this runs the sentex is
durably stored and indexed; the image is a cache of the derived half, and save! is
full of steps that can throw for reasons that say nothing about the record — a full
disk, a section that will not map. Letting one out would take down the assert after
it succeeded, and skip the observation seams that come after this call, leaving an
incremental matcher's alpha memories permanently behind the store. So the failure is
logged and swallowed, the same posture close-dir! above takes for the close-path save,
and the image simply stays as stale as it was — which the next open catches and rebuilds
from.
roots-now is passed in rather than read here — the caller has the index and this
namespace does not depend on it.
Rewrite `cdir`'s index image if the live index has drifted past the threshold. **`cdir` is already canonical** — the KB carries the resolved path (`kb`'s `:snapshot-dir`), so nothing here calls `getCanonicalPath` and the write door pays no `realpath` per assert. A caller holding a directory as its user spelled it canonicalizes it once and keeps the answer; this is the wrong place to do it per call. **Called on the writer's thread, from the write door** (`kb/create-sentex`), because that is the only thread allowed to touch the columnar index (`disk/index_snapshot.clj`, "the cadence"). So the gate is asked per write, and what it costs is worth stating rather than guessing at: one read of the registry here, which is where a directory with no image writer registered stops — and then, for one that has, `due?`'s two property reads, the canonical path its state is keyed by, one map read and a clock. The caller reaches this at all only when its KB carries a `:snapshot-dir`, which is what keeps the whole of it off every other backend's write path. **A refresh that throws must not fail the write.** By the time this runs the sentex is durably stored and indexed; the image is a cache of the derived half, and `save!` is full of steps that can throw for reasons that say nothing about the record — a full disk, a section that will not map. Letting one out would take down the assert *after* it succeeded, and skip the observation seams that come after this call, leaving an incremental matcher's alpha memories permanently behind the store. So the failure is logged and swallowed, the same posture `close-dir!` above takes for the close-path save, and the image simply stays as stale as it was — which the next open catches and rebuilds from. `roots-now` is passed in rather than read here — the caller has the index and this namespace does not depend on it.
(opened dir)What is currently open for dir: the subset of #{:records :index :overlay-meta} this
JVM holds stores for. Empty when the directory has never been opened (or has been
closed).
What is currently open for `dir`: the subset of `#{:records :index :overlay-meta}` this
JVM holds stores for. Empty when the directory has never been opened (or has been
closed).(overlay-meta-for dir)The durable KvBackend holding a fork's record-level bookkeeping for dir
(vaelii.impl.overlay.mount). Opened only for a directory that actually hosts a
fork's overlay.
The durable `KvBackend` holding a fork's record-level bookkeeping for `dir` (`vaelii.impl.overlay.mount`). Opened only for a directory that actually hosts a fork's overlay.
(records-for dir)The durable RecordStore for dir. Opens no index — the record half alone is what
a RAM-index mode (:disk-memory, :disk-dense, :disk-columnar) wants durable.
The durable `RecordStore` for `dir`. Opens no index — the record half alone is what a RAM-index mode (`:disk-memory`, `:disk-dense`, `:disk-columnar`) wants durable.
(register-index-snapshot! dir save-fn)Register save-fn (a thunk) as dir's derived-index snapshot writer.
A derived index is not one of the components above — nothing here opens it, and it has
no file handle to close — but the image of it is written to this directory and has to
be written by whoever last held it, while the records it is stamped against are still
open. So it hangs off the directory's registry entry and runs at the front of
close-dir!, and on JVM shutdown through the durability daemon.
Idempotent per directory: open-kb runs for every KB constructed over a directory and
they all share one index, so the first registration is the one that stands.
Register `save-fn` (a thunk) as `dir`'s derived-index snapshot writer. A derived index is not one of the components above — nothing here opens it, and it has no file handle to close — but the *image* of it is written to this directory and has to be written by whoever last held it, while the records it is stamped against are still open. So it hangs off the directory's registry entry and runs at the front of `close-dir!`, and on JVM shutdown through the durability daemon. Idempotent per directory: `open-kb` runs for every KB constructed over a directory and they all share one index, so the first registration is the one that stands.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |