The default in-memory backends for the two storage protocols, selected at KB
construction. The engine above the protocols never touches a concrete store, so a
KB built on these runs the whole engine with no external dependency; the on-disk
backend (vaelii.impl.disk) is the durable alternative.
The record store implements RecordStore directly over maps. The index reuses
vaelii.impl.kv/KvIndexStore — the one trie/roots/index implementation — over a
MemoryKvBackend: one map keyed by the structured key vectors (equal vectors are
equal keys), holding a Long at each counter key and a set at each set key.
kv-intersect is a clojure.set/intersection; kv-members returns the stored set
by reference (no serialization, no copy).
Durable-within-the-JVM semantics by space number. Two KBs constructed over the
same :space number must share state, or the persistence/recovery tests (a second KB
restarted over the same databases) would find an empty store. A process-global
registry keyed by space number provides that: (memory-record-store {:space 15}) twice
returns records backed by one state atom, and by one handle counter beside it.
clear-records! / clear-index! empty a space. (State lives only for the life of
the JVM; the on-disk backend is what survives a process restart.)
Single-writer. Pure runs one writer (docs/storage.md, "The single-writer
contract"), so a store's state is one atom mutated by swap!; reads deref a snapshot
and are lock-free. Interleaved writers would not be serializable.
The default in-memory backends for the two storage protocols, selected at KB
construction. The engine above the protocols never touches a concrete store, so a
KB built on these runs the whole engine with no external dependency; the on-disk
backend (`vaelii.impl.disk`) is the durable alternative.
The record store implements `RecordStore` directly over maps. The index reuses
`vaelii.impl.kv/KvIndexStore` — the one trie/roots/index implementation — over a
`MemoryKvBackend`: one map keyed by the structured key vectors (equal vectors are
equal keys), holding a `Long` at each counter key and a set at each set key.
`kv-intersect` is a `clojure.set/intersection`; `kv-members` returns the stored set
by reference (no serialization, no copy).
**Durable-within-the-JVM semantics by space number.** Two KBs constructed over the
same `:space` number must share state, or the persistence/recovery tests (a second KB
restarted over the same databases) would find an empty store. A process-global
registry keyed by space number provides that: `(memory-record-store {:space 15})` twice
returns records backed by *one* state atom, and by one handle counter beside it.
`clear-records!` / `clear-index!` empty a space. (State lives only for the life of
the JVM; the on-disk backend is what survives a process restart.)
**Single-writer.** Pure runs one writer (docs/storage.md, "The single-writer
contract"), so a store's state is one atom mutated by `swap!`; reads deref a snapshot
and are lock-free. Interleaved *writers* would not be serializable.During a bulk load, {:state <the loaded backend's state atom> :txn <a volatile! holding a transient of its map>}. While bound, that backend's writes land on the
transient (assoc! — no per-op HAMT path copy) and its reads answer from the same
transient, so a read inside the load sees every write the load made. The whole load is
one persistent! at the end instead of a swap! per fact. nil (the default) leaves
every op on the persistent atom; a read outside a load pays one read of this var and,
inside one, an identical? check. Use with-bulk-writes for a distinct load.
The binding names the backend it is for, because a dynamic binding is per thread,
not per backend: a write this thread makes to any other MemoryKvBackend while the
load runs — a second KB's index, a chaining callback asserting elsewhere — lands on
that backend's own atom (txn-for), never on the loaded one's transient. A reader on
another thread reads the atom, which holds the state before the load.
The accumulator's life is the batch's, and no longer. with-bulk-writes clears
the volatile as it installs, so the binding a body conveyed somewhere — a future, a
lazy seq realized afterwards — finds nothing to write on and takes the atom, which is
where a write outside the batch belongs. A transient reached after its persistent!
throws.
During a bulk load, `{:state <the loaded backend's state atom> :txn <a volatile!
holding a transient of its map>}`. While bound, **that** backend's writes land on the
transient (`assoc!` — no per-op HAMT path copy) and its reads answer from the same
transient, so a read inside the load sees every write the load made. The whole load is
one `persistent!` at the end instead of a `swap!` per fact. nil (the default) leaves
every op on the persistent atom; a read outside a load pays one read of this var and,
inside one, an `identical?` check. Use `with-bulk-writes` for a distinct load.
The binding names the backend it is for, because a dynamic binding is per *thread*,
not per backend: a write this thread makes to any other `MemoryKvBackend` while the
load runs — a second KB's index, a chaining callback asserting elsewhere — lands on
that backend's own atom (`txn-for`), never on the loaded one's transient. A reader on
another thread reads the atom, which holds the state before the load.
**The accumulator's life is the batch's, and no longer.** `with-bulk-writes` clears
the volatile as it installs, so the binding a body conveyed somewhere — a future, a
lazy seq realized afterwards — finds nothing to write on and takes the atom, which is
where a write outside the batch belongs. A transient reached after its `persistent!`
throws.(bulk-writes* bk f)with-bulk-writes' body as a thunk — the macro is a wrapper over this, so the
accumulator's whole life is one function's local and nothing about it is spliced into
a caller's code.
Three steps, in this order, and each is what makes the batch's accumulator private to the batch:
base, and the transient is built off that
value. base is what the install is checked against, so the batch knows what it
was accumulating over rather than assuming;persistent!-ed
transient that would throw;base, not a reset!. The atom is
per space and shared by every index store over that space, so a reset! would
silently discard anything that reached it while the batch ran — a second bulk load
stacked over this one being the way to get there without two threads. Under the
single-writer contract nothing does, which is exactly why the check is affordable:
it is one CAS per batch, and it fires only where an overwrite would lose data.A failed install is :stacked-batch. It is raised only when the body itself
returned — a body that threw has a failure of its own to report, and replacing it with
this one would hide it.
`with-bulk-writes`' body as a thunk — the macro is a wrapper over this, so the accumulator's whole life is one function's local and nothing about it is spliced into a caller's code. Three steps, in this order, and each is what makes the batch's accumulator **private** to the batch: - the state map is read **once**, into `base`, and the transient is built off that value. `base` is what the install is checked against, so the batch knows what it was accumulating over rather than assuming; - the volatile is cleared before the install, so the binding a body conveyed elsewhere (a future, a lazy seq realized later) finds no accumulator and takes the atom — the correct home for a write outside the batch, and not a `persistent!`-ed transient that would throw; - the install is a **compare-and-set** against `base`, not a `reset!`. The atom is per *space* and shared by every index store over that space, so a `reset!` would silently discard anything that reached it while the batch ran — a second bulk load stacked over this one being the way to get there without two threads. Under the single-writer contract nothing does, which is exactly why the check is affordable: it is one CAS per batch, and it fires only where an overwrite would lose data. A failed install is `:stacked-batch`. It is raised only when the body itself returned — a body that threw has a failure of its own to report, and replacing it with this one would hide it.
(drop-index-space! space)Forget the derived index state held under space — core/close!'s release of the RAM
index a disk-backed KB derived, keyed by its directory
(vaelii.impl.kb/derived-index-space), so a process opening durable KBs in a loop does
not keep one derived index per directory it has finished with. A no-op for a space
nothing holds — every pure in-RAM KB keeps its space, which is its store rather than a
derived cache. Returns true when an entry was dropped.
Forget the derived index state held under `space` — `core/close!`'s release of the RAM index a disk-backed KB derived, keyed by its directory (`vaelii.impl.kb/derived-index-space`), so a process opening durable KBs in a loop does not keep one derived index per directory it has finished with. A no-op for a `space` nothing holds — every pure in-RAM KB keeps its space, which is its store rather than a derived cache. Returns true when an entry was dropped.
(drop-record-space! space)Forget the records and the handle counter held under space, so a process that opens
a KB per private space does not keep every space it has finished with. A store already
holding the space keeps its atoms; a store opened on space afterwards starts empty.
Returns true when an entry was dropped.
Forget the records and the handle counter held under `space`, so a process that opens a KB per private space does not keep every space it has finished with. A store already holding the space keeps its atoms; a store opened on `space` afterwards starts empty. Returns true when an entry was dropped.
(memory-index-store opts)An in-memory IndexStore — KvIndexStore over a MemoryKvBackend.
An in-memory `IndexStore` — `KvIndexStore` over a `MemoryKvBackend`.
(memory-kv-backend {:keys [space] :or {space 0}})An in-memory KvBackend. Only :space in opts matters (it selects the shared
state atom, so two index stores over the same space number share one map).
An in-memory `KvBackend`. Only `:space` in `opts` matters (it selects the shared state atom, so two index stores over the same space number share one map).
(memory-record-store {:keys [space] :or {space 0}})An in-memory RecordStore. Only :space in opts matters: it selects the shared
state atom and the shared handle counter, which have to be the same pair for two
stores over one space or the second would re-issue handles the first had already
written.
An in-memory `RecordStore`. Only `:space` in `opts` matters: it selects the shared state atom **and** the shared handle counter, which have to be the same pair for two stores over one space or the second would re-issue handles the first had already written.
(with-bulk-writes backend & body)Run body with backend's index writes accumulated on one TRANSIENT of its state
map, persisted back in a single step at the end — the write-side fast path for a bulk
load (millions of trie assoc!s with no per-op HAMT path copy, one persistent!
instead of a swap! per fact). A no-op wrapper unless backend is a MemoryKvBackend,
so a non-memory store (disk) just runs body on its own batched path. A read of
backend inside body sees the writes body made (*bulk-txn*); bulk-writes*,
which this delegates to, keeps the accumulator the batch's own.
Run `body` with `backend`'s index writes accumulated on one TRANSIENT of its state map, persisted back in a single step at the end — the write-side fast path for a bulk load (millions of trie `assoc!`s with no per-op HAMT path copy, one `persistent!` instead of a `swap!` per fact). A no-op wrapper unless `backend` is a MemoryKvBackend, so a non-memory store (disk) just runs `body` on its own batched path. A read of `backend` inside `body` sees the writes `body` made (`*bulk-txn*`); `bulk-writes*`, which this delegates to, keeps the accumulator the batch's own.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |