Audit-chain verification for :crypto-hash? true scriptum branches.
verify-chain walks the linear sequence of Lucene commit generations
for a branch, asks the underlying BranchIndexWriter.verifyCommit to
recompute each commit's segment-file hashes against the stored
merkle metadata, and reports any divergence.
The protocol shape (IAuditable, -merkle-root,
-recompute-merkle-root) and the result-map vocabulary
({:status :ok|:mismatch|:unsupported|:advisory|:incomplete}) are
intentionally identical to the rest of the replikativ index libs
(datahike.index.audit, stratum.audit, proximum.audit). Bridges in
datahike pass results through without translation.
Audit-chain verification for `:crypto-hash? true` scriptum branches.
`verify-chain` walks the linear sequence of Lucene commit generations
for a branch, asks the underlying `BranchIndexWriter.verifyCommit` to
recompute each commit's segment-file hashes against the stored
merkle metadata, and reports any divergence.
The protocol shape (`IAuditable`, `-merkle-root`,
`-recompute-merkle-root`) and the result-map vocabulary
(`{:status :ok|:mismatch|:unsupported|:advisory|:incomplete}`) are
intentionally identical to the rest of the replikativ index libs
(datahike.index.audit, stratum.audit, proximum.audit). Bridges in
datahike pass results through without translation.COW branching semantics on top of Apache Lucene.
Provides fast forking (~3-5ms), structural sharing of immutable segments, branch-isolated indexing/searching, snapshot retention, and explicit GC.
Key concepts:
COW branching semantics on top of Apache Lucene. Provides fast forking (~3-5ms), structural sharing of immutable segments, branch-isolated indexing/searching, snapshot retention, and explicit GC. Key concepts: - Writer: mutable handle to a branch (one per branch per JVM) - Snapshot: immutable DirectoryReader at a specific commit point - Branch: COW overlay sharing base segments with the trunk - GC: explicit cleanup of old snapshots respecting branch references
Konserve-backed storage for scriptum indices.
Konserve is the source of truth; a local directory is a derived cache that
may be deleted at any time. This is proximum's dual-storage model
(proximum.vectors) applied to Lucene, and Lucene fits it better than
vectors do: segment files are WRITE-ONCE, so a cached file is valid forever
and never needs invalidating.
BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The
path-based design encodes a branch as branches/<name>/ and enumerates
branches with newDirectoryStream, which makes the filesystem the branch
registry — the role konserve is supposed to hold. Here a branch is
[:scriptum :manifest <branch>] -> <commit address>
which is the ONE mutable cell. Everything it reaches is immutable and content-addressed:
[:scriptum :snapshot <address>] -> {:files {lucene-filename -> address}
:parents [<address> ...]}
[:scriptum :blob <address>] -> the bytes
Three consequences, each verified by the tests in this namespace:
That third point removes the reason BranchAwareMergePolicy existed, the
second removes BranchedDirectory's base/overlay composition, and reachability
from the set of live manifests removes BranchDeletionPolicy's ref-counting.
CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per
branch, readers unconstrained. Lucene's own write.lock lives in the
per-branch view, so a second writer fails loudly with
LockObtainFailedException while writers on different branches proceed in
parallel.
THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and cache is a
caller-supplied string with no enforced relationship to branch identity. Two
writers on one branch with DIFFERENT cache paths — a per-process temp cache,
a container remount, a symlink — both succeed, on one machine, silently.
Within a single cache path it is solid: NativeFSLockFactory keeps a
process-wide held set (so a second writer in this JVM is refused by name) and
an OS advisory lock the kernel releases on abnormal exit (so another process
on the same machine is refused too). Nothing spans machines: the lock file is
in a local cache, so there is no shared file to contend on.
ACROSS RUNTIMES that is only half true, and the halves differ:
DIFFERENT branches are safe by construction, and this is the point of the manifest. Two writers on two branches touch disjoint keys — one manifest each, and blobs whose keys are content hashes, so a segment both happen to write is the same key holding the same bytes. Nothing needs coordinating because nothing is shared.
The SAME branch is NOT protected. write.lock lives in the local cache,
which is per-machine, so two processes both open a writer, both commit, and
the manifest write is last-writer-wins: the loser's segments are silently
orphaned. The manifest is written unconditionally, which is the same shape
as datahike's branch head (datahike#878) and carries the same verdict.
RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not
close it, because a deploy or container replacement still runs two
environments at once. Nor does a single serialized stream suffice on its
own: Lambda FREEZES an environment rather than terminating it, so a thawed
one holds a manifest atom, a Lucene writer and a /tmp cache from before
another environment advanced the branch. Open and close a writer per
invocation rather than caching one, or that stale writer derives its next
manifest from a superseded one. (The view cache is safe either way — a
stale entry is repaired by inode on first touch; see link-into-view!.)
The fix is a compare-and-set on the manifest. konserve-s3 already
implements put-object-conditional, so on S3 the primitive exists and is
opt-in via :config {:optimistic-locking-retries n} — but taking it needs
sync restructured to RETRY from a re-read manifest rather than write one
it computed from a cached value, and the GC guard is in-process regardless,
so a collection on one machine still cannot see another's in-flight blobs.
Until both land, treat multi-writer as unsupported: one writer per branch,
in one JVM.
THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather
than asserted here — BaseDirectoryTestCase, via scriptum.tck-runner. It
is worth running after any change to this namespace: the exception types, the
create/delete edge cases, and the listAll-under-concurrent-writes race were
all found by it and none of them by hand.
Konserve-backed storage for scriptum indices.
Konserve is the source of truth; a local directory is a derived cache that
may be deleted at any time. This is proximum's dual-storage model
(`proximum.vectors`) applied to Lucene, and Lucene fits it better than
vectors do: segment files are WRITE-ONCE, so a cached file is valid forever
and never needs invalidating.
BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The
path-based design encodes a branch as `branches/<name>/` and enumerates
branches with `newDirectoryStream`, which makes the filesystem the branch
registry — the role konserve is supposed to hold. Here a branch is
[:scriptum :manifest <branch>] -> <commit address>
which is the ONE mutable cell. Everything it reaches is immutable and
content-addressed:
[:scriptum :snapshot <address>] -> {:files {lucene-filename -> address}
:parents [<address> ...]}
[:scriptum :blob <address>] -> the bytes
Three consequences, each verified by the tests in this namespace:
1. Forking is copying a manifest. No bytes move.
2. Segments shared between branches are ONE blob, because the address is the
content hash. Locally they are one INODE too (the per-branch view is made
of hard links into a content-addressed pool), so shared segments occupy
memory once when mmap'd.
3. Merging is branch-local. A merge writes new blobs under new addresses and
leaves the old ones for whoever still references them.
That third point removes the reason `BranchAwareMergePolicy` existed, the
second removes `BranchedDirectory`'s base/overlay composition, and reachability
from the set of live manifests removes `BranchDeletionPolicy`'s ref-counting.
CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per
branch, readers unconstrained. Lucene's own `write.lock` lives in the
per-branch view, so a second writer fails loudly with
LockObtainFailedException while writers on different branches proceed in
parallel.
THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and `cache` is a
caller-supplied string with no enforced relationship to branch identity. Two
writers on one branch with DIFFERENT cache paths — a per-process temp cache,
a container remount, a symlink — both succeed, on one machine, silently.
Within a single cache path it is solid: NativeFSLockFactory keeps a
process-wide held set (so a second writer in this JVM is refused by name) and
an OS advisory lock the kernel releases on abnormal exit (so another process
on the same machine is refused too). Nothing spans machines: the lock file is
in a local cache, so there is no shared file to contend on.
ACROSS RUNTIMES that is only half true, and the halves differ:
- DIFFERENT branches are safe by construction, and this is the point of the
manifest. Two writers on two branches touch disjoint keys — one manifest
each, and blobs whose keys are content hashes, so a segment both happen to
write is the same key holding the same bytes. Nothing needs coordinating
because nothing is shared.
- The SAME branch is NOT protected. `write.lock` lives in the local cache,
which is per-machine, so two processes both open a writer, both commit, and
the manifest write is last-writer-wins: the loser's segments are silently
orphaned. The manifest is written unconditionally, which is the same shape
as datahike's branch head (datahike#878) and carries the same verdict.
RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not
close it, because a deploy or container replacement still runs two
environments at once. Nor does a single serialized stream suffice on its
own: Lambda FREEZES an environment rather than terminating it, so a thawed
one holds a manifest atom, a Lucene writer and a `/tmp` cache from before
another environment advanced the branch. Open and close a writer per
invocation rather than caching one, or that stale writer derives its next
manifest from a superseded one. (The view cache is safe either way — a
stale entry is repaired by inode on first touch; see `link-into-view!`.)
The fix is a compare-and-set on the manifest. `konserve-s3` already
implements `put-object-conditional`, so on S3 the primitive exists and is
opt-in via `:config {:optimistic-locking-retries n}` — but taking it needs
`sync` restructured to RETRY from a re-read manifest rather than write one
it computed from a cached value, and the GC guard is in-process regardless,
so a collection on one machine still cannot see another's in-flight blobs.
Until both land, treat multi-writer as unsupported: one writer per branch,
in one JVM.
THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather
than asserted here — `BaseDirectoryTestCase`, via `scriptum.tck-runner`. It
is worth running after any change to this namespace: the exception types, the
create/delete edge cases, and the `listAll`-under-concurrent-writes race were
all found by it and none of them by hand.Durable metadata index backed by persistent-sorted-set + konserve.
Provides O(log n) lookup of commit generations by custom metadata keys, with lazy loading from disk (O(1) startup) and incremental updates.
Each entry is {:branch b :key k :value v :generation g}. Sorted by [:branch :key :value] enabling efficient exact and floor queries.
Every store call is {:sync? true}. It used to block on konserve's async
channel with <!!, which measured 8.6x slower per write on the same
filestore — a flush issues eight of them, so it dominated commit cost for any
index carrying metadata. scriptum.konserve was already synchronous
throughout; this namespace was the outlier.
Durable metadata index backed by persistent-sorted-set + konserve.
Provides O(log n) lookup of commit generations by custom metadata keys,
with lazy loading from disk (O(1) startup) and incremental updates.
Each entry is {:branch b :key k :value v :generation g}.
Sorted by [:branch :key :value] enabling efficient exact and floor queries.
Every store call is `{:sync? true}`. It used to block on konserve's async
channel with `<!!`, which measured 8.6x slower per write on the same
filestore — a flush issues eight of them, so it dominated commit cost for any
index carrying metadata. `scriptum.konserve` was already synchronous
throughout; this namespace was the outlier.Yggdrasil adapter for scriptum COW indexes.
Uses VALUE SEMANTICS: mutating operations (branch!, checkout, etc.) return new ScriptumSystem values. The underlying BranchIndexWriter instances are mutable (Lucene writers), but the system structure (which writers exist, which is current) is immutable.
Snapshot IDs are UUIDs stored in commit user-data.
Yggdrasil adapter for scriptum COW indexes. Uses VALUE SEMANTICS: mutating operations (branch!, checkout, etc.) return new ScriptumSystem values. The underlying BranchIndexWriter instances are mutable (Lucene writers), but the system structure (which writers exist, which is current) is immutable. Snapshot IDs are UUIDs stored in commit user-data.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |