Liking cljdoc? Tell your friends :D

scriptum.audit

Audit-chain verification for :crypto-hash? true scriptum branches.

verify-chain walks the linear sequence of Lucene commit generations for a branch, asks the underlying BranchIndexWriter.verifyCommit to recompute each commit's segment-file hashes against the stored merkle metadata, and reports any divergence.

The protocol shape (IAuditable, -merkle-root, -recompute-merkle-root) and the result-map vocabulary ({:status :ok|:mismatch|:unsupported|:advisory|:incomplete}) are intentionally identical to the rest of the replikativ index libs (datahike.index.audit, stratum.audit, proximum.audit). Bridges in datahike pass results through without translation.

Audit-chain verification for `:crypto-hash? true` scriptum branches.

`verify-chain` walks the linear sequence of Lucene commit generations
for a branch, asks the underlying `BranchIndexWriter.verifyCommit` to
recompute each commit's segment-file hashes against the stored
merkle metadata, and reports any divergence.

The protocol shape (`IAuditable`, `-merkle-root`,
`-recompute-merkle-root`) and the result-map vocabulary
(`{:status :ok|:mismatch|:unsupported|:advisory|:incomplete}`) are
intentionally identical to the rest of the replikativ index libs
(datahike.index.audit, stratum.audit, proximum.audit). Bridges in
datahike pass results through without translation.
raw docstring

scriptum.core

COW branching semantics on top of Apache Lucene.

Provides fast forking (~3-5ms), structural sharing of immutable segments, branch-isolated indexing/searching, snapshot retention, and explicit GC.

Key concepts:

  • Writer: mutable handle to a branch (one per branch per JVM)
  • Snapshot: immutable DirectoryReader at a specific commit point
  • Branch: COW overlay sharing base segments with the trunk
  • GC: explicit cleanup of old snapshots respecting branch references
COW branching semantics on top of Apache Lucene.

Provides fast forking (~3-5ms), structural sharing of immutable segments,
branch-isolated indexing/searching, snapshot retention, and explicit GC.

Key concepts:
- Writer: mutable handle to a branch (one per branch per JVM)
- Snapshot: immutable DirectoryReader at a specific commit point
- Branch: COW overlay sharing base segments with the trunk
- GC: explicit cleanup of old snapshots respecting branch references
raw docstring

scriptum.konserve

Konserve-backed storage for scriptum indices.

Konserve is the source of truth; a local directory is a derived cache that may be deleted at any time. This is proximum's dual-storage model (proximum.vectors) applied to Lucene, and Lucene fits it better than vectors do: segment files are WRITE-ONCE, so a cached file is valid forever and never needs invalidating.

BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The path-based design encodes a branch as branches/<name>/ and enumerates branches with newDirectoryStream, which makes the filesystem the branch registry — the role konserve is supposed to hold. Here a branch is

[:scriptum :manifest <branch>]  ->  <commit address>

which is the ONE mutable cell. Everything it reaches is immutable and content-addressed:

[:scriptum :snapshot <address>] ->  {:files {lucene-filename -> address}
                                     :parents [<address> ...]}
[:scriptum :blob <address>]     ->  the bytes

Three consequences, each verified by the tests in this namespace:

  1. Forking is copying a manifest. No bytes move.
  2. Segments shared between branches are ONE blob, because the address is the content hash. Locally they are one INODE too (the per-branch view is made of hard links into a content-addressed pool), so shared segments occupy memory once when mmap'd.
  3. Merging is branch-local. A merge writes new blobs under new addresses and leaves the old ones for whoever still references them.

That third point removes the reason BranchAwareMergePolicy existed, the second removes BranchedDirectory's base/overlay composition, and reachability from the set of live manifests removes BranchDeletionPolicy's ref-counting.

CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per branch, readers unconstrained. Lucene's own write.lock lives in the per-branch view, so a second writer fails loudly with LockObtainFailedException while writers on different branches proceed in parallel.

THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and cache is a caller-supplied string with no enforced relationship to branch identity. Two writers on one branch with DIFFERENT cache paths — a per-process temp cache, a container remount, a symlink — both succeed, on one machine, silently. Within a single cache path it is solid: NativeFSLockFactory keeps a process-wide held set (so a second writer in this JVM is refused by name) and an OS advisory lock the kernel releases on abnormal exit (so another process on the same machine is refused too). Nothing spans machines: the lock file is in a local cache, so there is no shared file to contend on.

ACROSS RUNTIMES that is only half true, and the halves differ:

  • DIFFERENT branches are safe by construction, and this is the point of the manifest. Two writers on two branches touch disjoint keys — one manifest each, and blobs whose keys are content hashes, so a segment both happen to write is the same key holding the same bytes. Nothing needs coordinating because nothing is shared.

  • The SAME branch is NOT protected. write.lock lives in the local cache, which is per-machine, so two processes both open a writer, both commit, and the manifest write is last-writer-wins: the loser's segments are silently orphaned. The manifest is written unconditionally, which is the same shape as datahike's branch head (datahike#878) and carries the same verdict.

    RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not close it, because a deploy or container replacement still runs two environments at once. Nor does a single serialized stream suffice on its own: Lambda FREEZES an environment rather than terminating it, so a thawed one holds a manifest atom, a Lucene writer and a /tmp cache from before another environment advanced the branch. Open and close a writer per invocation rather than caching one, or that stale writer derives its next manifest from a superseded one. (The view cache is safe either way — a stale entry is repaired by inode on first touch; see link-into-view!.)

    The fix is a compare-and-set on the manifest. konserve-s3 already implements put-object-conditional, so on S3 the primitive exists and is opt-in via :config {:optimistic-locking-retries n} — but taking it needs sync restructured to RETRY from a re-read manifest rather than write one it computed from a cached value, and the GC guard is in-process regardless, so a collection on one machine still cannot see another's in-flight blobs. Until both land, treat multi-writer as unsupported: one writer per branch, in one JVM.

THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather than asserted here — BaseDirectoryTestCase, via scriptum.tck-runner. It is worth running after any change to this namespace: the exception types, the create/delete edge cases, and the listAll-under-concurrent-writes race were all found by it and none of them by hand.

Konserve-backed storage for scriptum indices.

Konserve is the source of truth; a local directory is a derived cache that
may be deleted at any time. This is proximum's dual-storage model
(`proximum.vectors`) applied to Lucene, and Lucene fits it better than
vectors do: segment files are WRITE-ONCE, so a cached file is valid forever
and never needs invalidating.

BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The
path-based design encodes a branch as `branches/<name>/` and enumerates
branches with `newDirectoryStream`, which makes the filesystem the branch
registry — the role konserve is supposed to hold. Here a branch is

    [:scriptum :manifest <branch>]  ->  <commit address>

which is the ONE mutable cell. Everything it reaches is immutable and
content-addressed:

    [:scriptum :snapshot <address>] ->  {:files {lucene-filename -> address}
                                         :parents [<address> ...]}
    [:scriptum :blob <address>]     ->  the bytes

Three consequences, each verified by the tests in this namespace:

1. Forking is copying a manifest. No bytes move.
2. Segments shared between branches are ONE blob, because the address is the
   content hash. Locally they are one INODE too (the per-branch view is made
   of hard links into a content-addressed pool), so shared segments occupy
   memory once when mmap'd.
3. Merging is branch-local. A merge writes new blobs under new addresses and
   leaves the old ones for whoever still references them.

That third point removes the reason `BranchAwareMergePolicy` existed, the
second removes `BranchedDirectory`'s base/overlay composition, and reachability
from the set of live manifests removes `BranchDeletionPolicy`'s ref-counting.

CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per
branch, readers unconstrained. Lucene's own `write.lock` lives in the
per-branch view, so a second writer fails loudly with
LockObtainFailedException while writers on different branches proceed in
parallel.

THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and `cache` is a
caller-supplied string with no enforced relationship to branch identity. Two
writers on one branch with DIFFERENT cache paths — a per-process temp cache,
a container remount, a symlink — both succeed, on one machine, silently.
Within a single cache path it is solid: NativeFSLockFactory keeps a
process-wide held set (so a second writer in this JVM is refused by name) and
an OS advisory lock the kernel releases on abnormal exit (so another process
on the same machine is refused too). Nothing spans machines: the lock file is
in a local cache, so there is no shared file to contend on.

ACROSS RUNTIMES that is only half true, and the halves differ:

- DIFFERENT branches are safe by construction, and this is the point of the
  manifest. Two writers on two branches touch disjoint keys — one manifest
  each, and blobs whose keys are content hashes, so a segment both happen to
  write is the same key holding the same bytes. Nothing needs coordinating
  because nothing is shared.
- The SAME branch is NOT protected. `write.lock` lives in the local cache,
  which is per-machine, so two processes both open a writer, both commit, and
  the manifest write is last-writer-wins: the loser's segments are silently
  orphaned. The manifest is written unconditionally, which is the same shape
  as datahike's branch head (datahike#878) and carries the same verdict.

  RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not
  close it, because a deploy or container replacement still runs two
  environments at once. Nor does a single serialized stream suffice on its
  own: Lambda FREEZES an environment rather than terminating it, so a thawed
  one holds a manifest atom, a Lucene writer and a `/tmp` cache from before
  another environment advanced the branch. Open and close a writer per
  invocation rather than caching one, or that stale writer derives its next
  manifest from a superseded one. (The view cache is safe either way — a
  stale entry is repaired by inode on first touch; see `link-into-view!`.)

  The fix is a compare-and-set on the manifest. `konserve-s3` already
  implements `put-object-conditional`, so on S3 the primitive exists and is
  opt-in via `:config {:optimistic-locking-retries n}` — but taking it needs
  `sync` restructured to RETRY from a re-read manifest rather than write one
  it computed from a cached value, and the GC guard is in-process regardless,
  so a collection on one machine still cannot see another's in-flight blobs.
  Until both land, treat multi-writer as unsupported: one writer per branch,
  in one JVM.

THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather
than asserted here — `BaseDirectoryTestCase`, via `scriptum.tck-runner`. It
is worth running after any change to this namespace: the exception types, the
create/delete edge cases, and the `listAll`-under-concurrent-writes race were
all found by it and none of them by hand.
raw docstring

scriptum.metadata

Durable metadata index backed by persistent-sorted-set + konserve.

Provides O(log n) lookup of commit generations by custom metadata keys, with lazy loading from disk (O(1) startup) and incremental updates.

Each entry is {:branch b :key k :value v :generation g}. Sorted by [:branch :key :value] enabling efficient exact and floor queries.

Every store call is {:sync? true}. It used to block on konserve's async channel with <!!, which measured 8.6x slower per write on the same filestore — a flush issues eight of them, so it dominated commit cost for any index carrying metadata. scriptum.konserve was already synchronous throughout; this namespace was the outlier.

Durable metadata index backed by persistent-sorted-set + konserve.

Provides O(log n) lookup of commit generations by custom metadata keys,
with lazy loading from disk (O(1) startup) and incremental updates.

Each entry is {:branch b :key k :value v :generation g}.
Sorted by [:branch :key :value] enabling efficient exact and floor queries.

Every store call is `{:sync? true}`. It used to block on konserve's async
channel with `<!!`, which measured 8.6x slower per write on the same
filestore — a flush issues eight of them, so it dominated commit cost for any
index carrying metadata. `scriptum.konserve` was already synchronous
throughout; this namespace was the outlier.
raw docstring

scriptum.yggdrasil

Yggdrasil adapter for scriptum COW indexes.

Uses VALUE SEMANTICS: mutating operations (branch!, checkout, etc.) return new ScriptumSystem values. The underlying BranchIndexWriter instances are mutable (Lucene writers), but the system structure (which writers exist, which is current) is immutable.

Snapshot IDs are UUIDs stored in commit user-data.

Yggdrasil adapter for scriptum COW indexes.

Uses VALUE SEMANTICS: mutating operations (branch!, checkout, etc.)
return new ScriptumSystem values. The underlying BranchIndexWriter
instances are mutable (Lucene writers), but the system structure
(which writers exist, which is current) is immutable.

Snapshot IDs are UUIDs stored in commit user-data.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
Move to previous article
Move to next article
Ctrl+/Jump to the search field
× close