Konserve-backed storage for scriptum indices.
Konserve is the source of truth; a local directory is a derived cache that
may be deleted at any time. This is proximum's dual-storage model
(proximum.vectors) applied to Lucene, and Lucene fits it better than
vectors do: segment files are WRITE-ONCE, so a cached file is valid forever
and never needs invalidating.
BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The
path-based design encodes a branch as branches/<name>/ and enumerates
branches with newDirectoryStream, which makes the filesystem the branch
registry — the role konserve is supposed to hold. Here a branch is
[:scriptum :manifest <branch>] -> <commit address>
which is the ONE mutable cell. Everything it reaches is immutable and content-addressed:
[:scriptum :snapshot <address>] -> {:files {lucene-filename -> address}
:parents [<address> ...]}
[:scriptum :blob <address>] -> the bytes
Three consequences, each verified by the tests in this namespace:
That third point removes the reason BranchAwareMergePolicy existed, the
second removes BranchedDirectory's base/overlay composition, and reachability
from the set of live manifests removes BranchDeletionPolicy's ref-counting.
CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per
branch, readers unconstrained. Lucene's own write.lock lives in the
per-branch view, so a second writer fails loudly with
LockObtainFailedException while writers on different branches proceed in
parallel.
THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and cache is a
caller-supplied string with no enforced relationship to branch identity. Two
writers on one branch with DIFFERENT cache paths — a per-process temp cache,
a container remount, a symlink — both succeed, on one machine, silently.
Within a single cache path it is solid: NativeFSLockFactory keeps a
process-wide held set (so a second writer in this JVM is refused by name) and
an OS advisory lock the kernel releases on abnormal exit (so another process
on the same machine is refused too). Nothing spans machines: the lock file is
in a local cache, so there is no shared file to contend on.
ACROSS RUNTIMES that is only half true, and the halves differ:
DIFFERENT branches are safe by construction, and this is the point of the manifest. Two writers on two branches touch disjoint keys — one manifest each, and blobs whose keys are content hashes, so a segment both happen to write is the same key holding the same bytes. Nothing needs coordinating because nothing is shared.
The SAME branch is NOT protected. write.lock lives in the local cache,
which is per-machine, so two processes both open a writer, both commit, and
the manifest write is last-writer-wins: the loser's segments are silently
orphaned. The manifest is written unconditionally, which is the same shape
as datahike's branch head (datahike#878) and carries the same verdict.
RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not
close it, because a deploy or container replacement still runs two
environments at once. Nor does a single serialized stream suffice on its
own: Lambda FREEZES an environment rather than terminating it, so a thawed
one holds a manifest atom, a Lucene writer and a /tmp cache from before
another environment advanced the branch. Open and close a writer per
invocation rather than caching one, or that stale writer derives its next
manifest from a superseded one. (The view cache is safe either way — a
stale entry is repaired by inode on first touch; see link-into-view!.)
The fix is a compare-and-set on the manifest. konserve-s3 already
implements put-object-conditional, so on S3 the primitive exists and is
opt-in via :config {:optimistic-locking-retries n} — but taking it needs
sync restructured to RETRY from a re-read manifest rather than write one
it computed from a cached value, and the GC guard is in-process regardless,
so a collection on one machine still cannot see another's in-flight blobs.
Until both land, treat multi-writer as unsupported: one writer per branch,
in one JVM.
THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather
than asserted here — BaseDirectoryTestCase, via scriptum.tck-runner. It
is worth running after any change to this namespace: the exception types, the
create/delete edge cases, and the listAll-under-concurrent-writes race were
all found by it and none of them by hand.
Konserve-backed storage for scriptum indices.
Konserve is the source of truth; a local directory is a derived cache that
may be deleted at any time. This is proximum's dual-storage model
(`proximum.vectors`) applied to Lucene, and Lucene fits it better than
vectors do: segment files are WRITE-ONCE, so a cached file is valid forever
and never needs invalidating.
BRANCH IDENTITY LIVES IN A MANIFEST, NOT IN A DIRECTORY TREE. The
path-based design encodes a branch as `branches/<name>/` and enumerates
branches with `newDirectoryStream`, which makes the filesystem the branch
registry — the role konserve is supposed to hold. Here a branch is
[:scriptum :manifest <branch>] -> <commit address>
which is the ONE mutable cell. Everything it reaches is immutable and
content-addressed:
[:scriptum :snapshot <address>] -> {:files {lucene-filename -> address}
:parents [<address> ...]}
[:scriptum :blob <address>] -> the bytes
Three consequences, each verified by the tests in this namespace:
1. Forking is copying a manifest. No bytes move.
2. Segments shared between branches are ONE blob, because the address is the
content hash. Locally they are one INODE too (the per-branch view is made
of hard links into a content-addressed pool), so shared segments occupy
memory once when mmap'd.
3. Merging is branch-local. A merge writes new blobs under new addresses and
leaves the old ones for whoever still references them.
That third point removes the reason `BranchAwareMergePolicy` existed, the
second removes `BranchedDirectory`'s base/overlay composition, and reachability
from the set of live manifests removes `BranchDeletionPolicy`'s ref-counting.
CONCURRENCY, WITHIN ONE RUNTIME, follows konserve's contract: one writer per
branch, readers unconstrained. Lucene's own `write.lock` lives in the
per-branch view, so a second writer fails loudly with
LockObtainFailedException while writers on different branches proceed in
parallel.
THAT LOCK'S SCOPE IS THE CACHE DIRECTORY, NOT THE MACHINE, and `cache` is a
caller-supplied string with no enforced relationship to branch identity. Two
writers on one branch with DIFFERENT cache paths — a per-process temp cache,
a container remount, a symlink — both succeed, on one machine, silently.
Within a single cache path it is solid: NativeFSLockFactory keeps a
process-wide held set (so a second writer in this JVM is refused by name) and
an OS advisory lock the kernel releases on abnormal exit (so another process
on the same machine is refused too). Nothing spans machines: the lock file is
in a local cache, so there is no shared file to contend on.
ACROSS RUNTIMES that is only half true, and the halves differ:
- DIFFERENT branches are safe by construction, and this is the point of the
manifest. Two writers on two branches touch disjoint keys — one manifest
each, and blobs whose keys are content hashes, so a segment both happen to
write is the same key holding the same bytes. Nothing needs coordinating
because nothing is shared.
- The SAME branch is NOT protected. `write.lock` lives in the local cache,
which is per-machine, so two processes both open a writer, both commit, and
the manifest write is last-writer-wins: the loser's segments are silently
orphaned. The manifest is written unconditionally, which is the same shape
as datahike's branch head (datahike#878) and carries the same verdict.
RESERVED CONCURRENCY 1 IS NOT A FIX. It narrows the window; it does not
close it, because a deploy or container replacement still runs two
environments at once. Nor does a single serialized stream suffice on its
own: Lambda FREEZES an environment rather than terminating it, so a thawed
one holds a manifest atom, a Lucene writer and a `/tmp` cache from before
another environment advanced the branch. Open and close a writer per
invocation rather than caching one, or that stale writer derives its next
manifest from a superseded one. (The view cache is safe either way — a
stale entry is repaired by inode on first touch; see `link-into-view!`.)
The fix is a compare-and-set on the manifest. `konserve-s3` already
implements `put-object-conditional`, so on S3 the primitive exists and is
opt-in via `:config {:optimistic-locking-retries n}` — but taking it needs
`sync` restructured to RETRY from a re-read manifest rather than write one
it computed from a cached value, and the GC guard is in-process regardless,
so a collection on one machine still cannot see another's in-flight blobs.
Until both land, treat multi-writer as unsupported: one writer per branch,
in one JVM.
THE LUCENE CONTRACT is checked against Lucene's own conformance suite rather
than asserted here — `BaseDirectoryTestCase`, via `scriptum.tck-runner`. It
is worth running after any change to this namespace: the exception types, the
create/delete edge cases, and the `listAll`-under-concurrent-writes race were
all found by it and none of them by hand.How many segment blobs a single sync uploads at once.
BOUNDED FOR MEMORY, not for the connection pool — konserve-s3 uses the AWS
SDK's UrlConnectionHttpClient, which has no pool to exhaust, so unbounded
concurrency would not fail there. What it would cost is heap: each in-flight
upload holds a konserve streaming buffer, 1 MB by default, so the 32
simultaneous PUTs a 35-blob commit was measured issuing are 32 MB of transient
buffers. On the 512 MB Lambda this namespace is written for, that matters.
Past the JVM keep-alive cache (http.maxConnections, 5 by default) the extra
sockets are also handshake churn rather than reuse.
pmap alone is not a bound: it realizes a chunk ahead, which is where 32 came
from. Enough to hide latency — the win is round trips overlapped, and beyond a
dozen or so there is nothing left to hide.
How many segment blobs a single `sync` uploads at once. BOUNDED FOR MEMORY, not for the connection pool — konserve-s3 uses the AWS SDK's `UrlConnectionHttpClient`, which has no pool to exhaust, so unbounded concurrency would not fail there. What it would cost is heap: each in-flight upload holds a konserve streaming buffer, 1 MB by default, so the 32 simultaneous PUTs a 35-blob commit was measured issuing are 32 MB of transient buffers. On the 512 MB Lambda this namespace is written for, that matters. Past the JVM keep-alive cache (`http.maxConnections`, 5 by default) the extra sockets are also handshake churn rather than reuse. `pmap` alone is not a bound: it realizes a chunk ahead, which is where 32 came from. Enough to hide latency — the win is round trips overlapped, and beyond a dozen or so there is nothing left to hide.
(branch-exists? store branch)Does branch have a manifest? Existence of a branch IS existence of its
manifest, so this is a lookup rather than an enumeration.
Exposed so a caller can refuse BEFORE doing work it would have to undo — the
store-backed fork committed its parent first and only then discovered the
target, leaving a commit point behind on every refused attempt.
THE REGISTRY COUNTS, NOT JUST THE MANIFEST. A branch that has been opened but
not yet committed is in the registry with no manifest — konserve-directory
registers at open — and testing only the manifest declared such a branch
available as a fork target. fork then committed the source, overwrote that
branch's pointer with the source's snapshot, and only afterwards failed
opening the forked writer on its held write.lock. The call reported failure
having already moved someone else's branch.
Does `branch` have a manifest? Existence of a branch IS existence of its manifest, so this is a lookup rather than an enumeration. Exposed so a caller can refuse BEFORE doing work it would have to undo — the store-backed `fork` committed its parent first and only then discovered the target, leaving a commit point behind on every refused attempt. THE REGISTRY COUNTS, NOT JUST THE MANIFEST. A branch that has been opened but not yet committed is in the registry with no manifest — `konserve-directory` registers at open — and testing only the manifest declared such a branch available as a fork target. `fork` then committed the source, overwrote that branch's pointer with the source's snapshot, and only afterwards failed opening the forked writer on its held `write.lock`. The call reported failure having already moved someone else's branch.
(branch-snapshot store branch)The snapshot address branch points at, or nil for a branch with no commit.
THE ONE MUTABLE CELL. Every other key in the store is immutable and content-
addressed, which is what makes a snapshot address something a caller can hold:
it names an index state that cannot change under them, where a branch name
cannot. datahike's secondary-index key-map wants this rather than the branch,
for the same reason proximum's carries a :commit-id.
THE ONE CHOKE POINT for an unmigrated v1 cell, which is why it throws rather than passing the value along: a v1 cell holds the file map itself, and a map used as a snapshot address resolves to nothing. Every read of a branch goes through here, so catching it once catches it everywhere.
The snapshot address `branch` points at, or nil for a branch with no commit. THE ONE MUTABLE CELL. Every other key in the store is immutable and content- addressed, which is what makes a snapshot address something a caller can hold: it names an index state that cannot change under them, where a branch name cannot. datahike's secondary-index key-map wants this rather than the branch, for the same reason proximum's carries a `:commit-id`. THE ONE CHOKE POINT for an unmigrated v1 cell, which is why it throws rather than passing the value along: a v1 cell holds the file map itself, and a map used as a snapshot address resolves to nothing. Every read of a branch goes through here, so catching it once catches it everywhere.
(branches store)Every branch of this index, from the registry.
A REGISTRY rather than a scan of the keyspace. The manifest key encodes its
branch name, so branches could be derived by filtering k/keys — and were —
but k/keys is not a listing: konserve OPENS AND READS EVERY BLOB to recover
its key. That made enumerating two branches cost one read per segment in the
store. Measured on a 155-key index it was 20.4 ms against 0.23 ms for a single
lookup, ~90x, and on an object store it is one GET per object.
proximum (:branches) and stratum ([:datasets :branches]) both keep the
same registry; this was the outlier.
Every branch of this index, from the registry. A REGISTRY rather than a scan of the keyspace. The manifest key encodes its branch name, so branches could be derived by filtering `k/keys` — and were — but `k/keys` is not a listing: konserve OPENS AND READS EVERY BLOB to recover its key. That made enumerating two branches cost one read per segment in the store. Measured on a 155-key index it was 20.4 ms against 0.23 ms for a single lookup, ~90x, and on an object store it is one GET per object. proximum (`:branches`) and stratum (`[:datasets :branches]`) both keep the same registry; this was the outlier.
Where the branch registry lives: a set of branch names.
Where the branch registry lives: a set of branch names.
(delete-branch! store branch)Forget branch. Blobs it referenced survive until gc! finds them
unreachable from every remaining manifest.
Forget `branch`. Blobs it referenced survive until `gc!` finds them unreachable from every remaining manifest.
(ensure-format! store)Stamp a fresh store with this layout, or refuse one we cannot read.
THE REFUSAL IS THE POINT. Without it, a store written by a different scriptum
is read as though it were this one, and fails as corruption somewhere far from
the cause. That has already happened once in miniature: the blob address
function changed from hasch/uuid to ContentHash during development, and
nothing could distinguish a store written before from one written after —
both produce valid-looking UUIDs for the same bytes, so the only symptom was
a manifest naming blobs that were not there.
NO MIGRATION, DELIBERATELY. No earlier layout was ever released — no released
scriptum contains this namespace at all, and the version stamp itself postdates
every build that exists — so the only older stores are development ones, which
are cheaper to discard than to convert. That exemption ends at the first
publication, which is why the layout was settled before it rather than after. Nothing here migrates path-based
(BranchedDirectory) indices either; that is a different storage model and a
separate problem.
Writing the converter anyway was actively harmful: it read the branch REGISTRY
to decide what to convert, and an incomplete registry is exactly what this
repository's earlier missing-GC-root bug produced. Branches it missed kept
their v1 maps, the stamp then recorded the store as converted so it never
retried, those branches read as empty, and the next gc! swept their blobs.
Refusing has none of that surface.
A store is fresh iff nothing has registered a branch in it — register-branch!
runs immediately after this on the first open, so the registry existing means
the store predates this layout. One extra key read, and only when unstamped.
Costs one key read per Directory open, next to the one register-branch!
already does — nothing on the listAll path. Returns the store's version.
Stamp a fresh store with this layout, or refuse one we cannot read. THE REFUSAL IS THE POINT. Without it, a store written by a different scriptum is read as though it were this one, and fails as corruption somewhere far from the cause. That has already happened once in miniature: the blob address function changed from `hasch/uuid` to `ContentHash` during development, and nothing could distinguish a store written before from one written after — both produce valid-looking UUIDs for the same bytes, so the only symptom was a manifest naming blobs that were not there. NO MIGRATION, DELIBERATELY. No earlier layout was ever released — no released scriptum contains this namespace at all, and the version stamp itself postdates every build that exists — so the only older stores are development ones, which are cheaper to discard than to convert. That exemption ends at the first publication, which is why the layout was settled before it rather than after. Nothing here migrates path-based (`BranchedDirectory`) indices either; that is a different storage model and a separate problem. Writing the converter anyway was actively harmful: it read the branch REGISTRY to decide what to convert, and an incomplete registry is exactly what this repository's earlier missing-GC-root bug produced. Branches it missed kept their v1 maps, the stamp then recorded the store as converted so it never retried, those branches read as empty, and the next `gc!` swept their blobs. Refusing has none of that surface. A store is fresh iff nothing has registered a branch in it — `register-branch!` runs immediately after this on the first open, so the registry existing means the store predates this layout. One extra key read, and only when unstamped. Costs one key read per Directory open, next to the one `register-branch!` already does — nothing on the `listAll` path. Returns the store's version.
(fork! store from to)Branch from as to: copy the manifest. O(1) — no segment bytes move, and
the two branches share every blob they have in common.
Branch `from` as `to`: copy the manifest. O(1) — no segment bytes move, and the two branches share every blob they have in common.
(fork-from-snapshot! store to address)Branch to at the index state address, which must not already exist.
fork! copies whatever the source branch names NOW; this names a specific
state, which is what a caller holding a snapshot address wants — datahike's
branch-from-key-map hands the key-map of an OLD commit, and forking the head
there would branch from the wrong place entirely.
Branch `to` at the index state `address`, which must not already exist. `fork!` copies whatever the source branch names NOW; this names a specific state, which is what a caller holding a snapshot address wants — datahike's `branch-from-key-map` hands the key-map of an OLD commit, and forking the head there would branch from the wrong place entirely.
Where the store records which manifest layout it is in.
Where the store records which manifest layout it is in.
The manifest layout this code writes and understands.
1 — [:scriptum :manifest <branch>] held the file map itself: a mutable cell
containing the whole tree.
2 — that cell holds an address, and the tree lives at
[:scriptum :snapshot <address>] addressed by the FILE MAP alone.
3 — the same, but the address covers the file map AND the parent it
descends from, and the value carries that parent: a commit hash
rather than a tree hash. Two commits with identical content in
different lineages no longer collide, and a head address covers
every ancestor as well as every segment.
The manifest layout this code writes and understands.
1 — `[:scriptum :manifest <branch>]` held the file map itself: a mutable cell
containing the whole tree.
2 — that cell holds an address, and the tree lives at
`[:scriptum :snapshot <address>]` addressed by the FILE MAP alone.
3 — the same, but the address covers the file map AND the parent it
descends from, and the value carries that parent: a commit hash
rather than a tree hash. Two commits with identical content in
different lineages no longer collide, and a head address covers
every ancestor as well as every segment.(gc! store store-id)(gc! store store-id ts)(gc! store store-id ts extra-snapshots)(gc! store store-id ts extra-snapshots extra-keys)Collect blobs no branch references any more.
Reachability from the live manifests IS the root set, which is why no
ref-counting deletion policy is needed. store-id must be the one every
writer on these bytes passes to konserve-directory — see there for why
agreeing matters and which way it fails — so the sweep can see their
in-flight writes.
Collection is EVENTUAL, not immediate. Write stamps have millisecond granularity and the sweep spares ties, so a blob written in the same millisecond as the call survives to the next cycle.
cutoff defaults to now and can only make a collection MORE conservative:
the sweep takes min(cutoff, safe-point), and the safe point never runs
ahead of now, so passing a later instant cannot force an earlier collection.
Pass one to hold back a collection ("nothing newer than X"), not to hurry it.
Synchronous, returning the set of collected keys.
It sweeps with {:sync? true} rather than taking konserve's channel with
<!!, which is what this did before. <!! DEADLOCKS inside a go block, and
gc! is reachable from a datahike writer that does run in async contexts — so
that was a latent hang, not a style choice. Staying synchronous also keeps the
call stack sync all the way up through scriptum.core/gc! to datahike's
secondary-index adapter, which has no async seam to thread this through.
extra-snapshots protects index states an EXTERNAL holder still references —
see reachable-snapshots. Anything embedding scriptum in a store it does not
own must pass them, or a state no branch names is collected out from under it.
NOTE for shared stores: this collects blobs no reachable snapshot names. A
reader on another machine pinned to an older snapshot can still be holding
one — and now has an address it can pass as extra-snapshots to say so.
Collect blobs no branch references any more.
Reachability from the live manifests IS the root set, which is why no
ref-counting deletion policy is needed. `store-id` must be the one every
writer on these bytes passes to `konserve-directory` — see there for why
agreeing matters and which way it fails — so the sweep can see their
in-flight writes.
Collection is EVENTUAL, not immediate. Write stamps have millisecond
granularity and the sweep spares ties, so a blob written in the same
millisecond as the call survives to the next cycle.
`cutoff` defaults to now and can only make a collection MORE conservative:
the sweep takes `min(cutoff, safe-point)`, and the safe point never runs
ahead of now, so passing a later instant cannot force an earlier collection.
Pass one to hold back a collection ("nothing newer than X"), not to hurry it.
Synchronous, returning the set of collected keys.
It sweeps with `{:sync? true}` rather than taking konserve's channel with
`<!!`, which is what this did before. `<!!` DEADLOCKS inside a go block, and
`gc!` is reachable from a datahike writer that does run in async contexts — so
that was a latent hang, not a style choice. Staying synchronous also keeps the
call stack sync all the way up through `scriptum.core/gc!` to datahike's
secondary-index adapter, which has no async seam to thread this through.
`extra-snapshots` protects index states an EXTERNAL holder still references —
see `reachable-snapshots`. Anything embedding scriptum in a store it does not
own must pass them, or a state no branch names is collected out from under it.
NOTE for shared stores: this collects blobs no reachable snapshot names. A
reader on another machine pinned to an older snapshot can still be holding
one — and now has an address it can pass as `extra-snapshots` to say so.(gc-cache! store cache)(gc-cache! store cache extra-snapshots)Delete pooled blobs no branch's manifest names, and views of branches that
are gone. Returns {:blobs n :views n}.
THE STORE COLLECTOR DOES NOT TOUCH THE CACHE, and the cache is where the
bytes actually sit on a machine. Measured on a merge-heavy workload: gc!
reclaimed 82% of the store and 0% of the pool, which held 73 blobs against 14
live addresses. On a long-running container that grows without bound; on AWS
Lambda, whose /tmp is 512 MB, it is a hard failure with a store a fraction
of the size.
Safe by construction, in a way the store collector is not: the pool is a DERIVED cache, so anything deleted here can be fetched again. The worst case is a re-download, never a dangling reference — which is why this needs no guard, no cutoff and no in-flight-write protection.
Nor does it disturb a running reader. Unlinking a mapped file is safe on POSIX — the inode outlives the directory entry for as long as anything maps it — and a live branch view holds a hard link to the same inode regardless.
A DEAD BRANCH'S VIEW KEEPS ITS write.lock, exempted exactly as the open-time
reconcile exempts it. The lock is Lucene's, not ours, and deleting one out
from under a live writer breaks the exclusion it provides: a writer whose
branch has been dropped from the registry — or deleted while it was open —
otherwise fails its next commit with NoSuchFileException on the lock itself.
Leaving one file behind is the cheaper mistake.
extra-snapshots must name the same held states passed to gc!. Without
them a snapshot view is reclaimed and its blobs re-downloaded on every call,
which is a cost bug rather than a correctness one — but a pointless one.
Same root set as gc!: reachability from the live manifests. Anything in the
pool whose name is not a live address is garbage, including the .tmp debris
of an interrupted materialization.
Delete pooled blobs no branch's manifest names, and views of branches that
are gone. Returns `{:blobs n :views n}`.
THE STORE COLLECTOR DOES NOT TOUCH THE CACHE, and the cache is where the
bytes actually sit on a machine. Measured on a merge-heavy workload: `gc!`
reclaimed 82% of the store and 0% of the pool, which held 73 blobs against 14
live addresses. On a long-running container that grows without bound; on AWS
Lambda, whose `/tmp` is 512 MB, it is a hard failure with a store a fraction
of the size.
Safe by construction, in a way the store collector is not: the pool is a
DERIVED cache, so anything deleted here can be fetched again. The worst case
is a re-download, never a dangling reference — which is why this needs no
guard, no cutoff and no in-flight-write protection.
Nor does it disturb a running reader. Unlinking a mapped file is safe on
POSIX — the inode outlives the directory entry for as long as anything maps
it — and a live branch view holds a hard link to the same inode regardless.
A DEAD BRANCH'S VIEW KEEPS ITS `write.lock`, exempted exactly as the open-time
reconcile exempts it. The lock is Lucene's, not ours, and deleting one out
from under a live writer breaks the exclusion it provides: a writer whose
branch has been dropped from the registry — or deleted while it was open —
otherwise fails its next commit with NoSuchFileException on the lock itself.
Leaving one file behind is the cheaper mistake.
`extra-snapshots` must name the same held states passed to `gc!`. Without
them a snapshot view is reclaimed and its blobs re-downloaded on every call,
which is a cost bug rather than a correctness one — but a pointless one.
Same root set as `gc!`: reachability from the live manifests. Anything in the
pool whose name is not a live address is garbage, including the `.tmp` debris
of an interrupted materialization.(konserve-directory store cache branch)(konserve-directory store cache branch store-id)(konserve-directory store cache branch store-id pending-parents)A Lucene Directory for branch, durable in store, read through an
mmap'd local cache under cache.
store-id keys konserve.gc-guard's safe point, and should be the store's
own :id — konserve's LOGICAL store identity, which is deliberately the same
across machines and backends holding the same store, and is NOT a name for
the bytes on this disk.
That distinction decides whether a collection is correct, and only one direction is dangerous:
So the id may be coarser than the physical store, never finer. Omitting it disables the guard, which is only safe on a store that is never collected.
A Lucene `Directory` for `branch`, durable in `store`, read through an mmap'd local cache under `cache`. `store-id` keys `konserve.gc-guard`'s safe point, and should be the store's own `:id` — konserve's LOGICAL store identity, which is deliberately the same across machines and backends holding the same store, and is NOT a name for the bytes on this disk. That distinction decides whether a collection is correct, and only one direction is dangerous: - Every writer on the SAME bytes passing the SAME id is the requirement. Get that wrong — two connections to one store made with different ids — and a sweep runs against a safe point that cannot see the other writer's in-flight blobs, and deletes what a manifest is about to reference. - Two SEPARATE stores sharing one id (a store and its replica) is merely conservative: each sweep is held back by the other's writers. Nothing is lost, though a continuously-written replica can hold a collection off. So the id may be coarser than the physical store, never finer. Omitting it disables the guard, which is only safe on a store that is never collected.
(manifest-branches store)Every branch with a manifest key, by KEYSPACE SCAN rather than the registry.
Expensive, and deliberately not on any read path — branches answers from
the registry. This is for repair-branches!, the one job where trusting the
registry is the bug, because rebuilding it is the point.
Every branch with a manifest key, by KEYSPACE SCAN rather than the registry. Expensive, and deliberately not on any read path — `branches` answers from the registry. This is for `repair-branches!`, the one job where trusting the registry is the bug, because rebuilding it is the point.
(manifest-key branch)The branch pointer. Holds a SNAPSHOT ADDRESS, not the file map — see
snapshot-key. Kept under this name because it is what a branch resolves
through, and renaming it would be a third layout.
The branch pointer. Holds a SNAPSHOT ADDRESS, not the file map — see `snapshot-key`. Kept under this name because it is what a branch resolves through, and renaming it would be a third layout.
(mark store)(mark store extra-snapshots)Every store key scriptum needs kept — the mark half of a mark-and-sweep.
EXPORTED BECAUSE AN EMBEDDER HAS TO CALL IT. When scriptum's blobs live in a
store it does not own — datahike's, via sec/mark-from-key-map — that store's
collector builds one whitelist from every index and sweeps everything else.
Leaving this inline in gc! meant such a caller had to re-derive the root set
by hand, and the two roots that are easy to miss are exactly the two already
missed once here: the branch registry and the format stamp. A swept registry
makes the next mark find no branches and take the whole index with it.
So this is the contract, in one place, and gc! is a caller of it like any
other.
IT COVERS THIS NAMESPACE ONLY. A scriptum.metadata index sharing the store
keeps its roots under bare keywords and every tree node under a raw UUID, so
nothing here can infer them — union scriptum.metadata/mark as well, or a
sweep from this whitelist alone deletes the metadata index outright. Same for
any other component on the store: allow-list means silence is deletion.
extra-snapshots names index states an external holder still references; see
reachable-snapshots. They are treated as HINTS: one that no longer exists is
ignored rather than fatal, because the holder is not scriptum and its list is
expected to lag. datahike's key-map is exactly where superseded addresses
collect, and making one stale entry stop collection for the whole store would
punish the caller mark was exported for. A dangling BRANCH pointer is the
opposite case — scriptum owns that, and it is corruption.
Superseded snapshots are deliberately NOT included; collecting them is what stops a long history accumulating one tree per commit.
Every store key scriptum needs kept — the mark half of a mark-and-sweep. EXPORTED BECAUSE AN EMBEDDER HAS TO CALL IT. When scriptum's blobs live in a store it does not own — datahike's, via `sec/mark-from-key-map` — that store's collector builds one whitelist from every index and sweeps everything else. Leaving this inline in `gc!` meant such a caller had to re-derive the root set by hand, and the two roots that are easy to miss are exactly the two already missed once here: the branch registry and the format stamp. A swept registry makes the next mark find no branches and take the whole index with it. So this is the contract, in one place, and `gc!` is a caller of it like any other. IT COVERS THIS NAMESPACE ONLY. A `scriptum.metadata` index sharing the store keeps its roots under bare keywords and every tree node under a raw UUID, so nothing here can infer them — union `scriptum.metadata/mark` as well, or a sweep from this whitelist alone deletes the metadata index outright. Same for any other component on the store: allow-list means silence is deletion. `extra-snapshots` names index states an external holder still references; see `reachable-snapshots`. They are treated as HINTS: one that no longer exists is ignored rather than fatal, because the holder is not scriptum and its list is expected to lag. datahike's key-map is exactly where superseded addresses collect, and making one stale entry stop collection for the whole store would punish the caller `mark` was exported for. A dangling BRANCH pointer is the opposite case — scriptum owns that, and it is corruption. Superseded snapshots are deliberately NOT included; collecting them is what stops a long history accumulating one tree per commit.
(normalize-parents parents)The canonical form of a parent set: distinct, nil-free, sorted by string.
Sorted because the address is computed over it and ContentHash serializes a
collection in iteration order — so an unordered set would give one commit two
addresses depending on how it happened to iterate.
The canonical form of a parent set: distinct, nil-free, sorted by string. Sorted because the address is computed over it and `ContentHash` serializes a collection in iteration order — so an unordered set would give one commit two addresses depending on how it happened to iterate.
(point-branch-at! store branch address)Make branch name the index state at address.
THE OPERATION THAT MAKES A SNAPSHOT ADDRESS WORTH HOLDING. snapshot-directory
can already read one, but nothing could turn one back into a writable branch —
so a caller holding an address (datahike's secondary-index key-map) could not
restore to it, and opening the branch silently gave whatever the branch had
moved on to instead. Primary and secondary then disagree with nothing
detecting it, which is the failure the immutable key-map was supposed to make
unrepresentable.
VERIFIED AFTER THE WRITE, for the same reason fork! is: the snapshot is old,
so the gc-guard cannot protect it — the guard spares what is written inside its
window, and this was written by some earlier commit. A collection that marked
before this pointer landed sweeps it regardless. Re-checking afterwards catches
that, and the pointer is removed rather than left dangling, since mark
resolves every branch pointer and one dangling branch stops collection for the
whole store.
A caller that needs the state to survive a concurrent collection must pass the
address as extra-snapshots to gc!; that is what it is for.
THIS DROPS WHATEVER branch NAMED BEFORE. The superseded snapshot becomes
unreachable and collectable — correct COW semantics, and the caller's business
to hold if they still want it. It also does not disturb a live writer on the
branch, because it cannot see one: reset a branch someone is writing and their
next commit derives from the state they held, not this one.
Returns the file map now at branch.
Make `branch` name the index state at `address`. THE OPERATION THAT MAKES A SNAPSHOT ADDRESS WORTH HOLDING. `snapshot-directory` can already read one, but nothing could turn one back into a writable branch — so a caller holding an address (datahike's secondary-index key-map) could not restore to it, and opening the branch silently gave whatever the branch had moved on to instead. Primary and secondary then disagree with nothing detecting it, which is the failure the immutable key-map was supposed to make unrepresentable. VERIFIED AFTER THE WRITE, for the same reason `fork!` is: the snapshot is old, so the gc-guard cannot protect it — the guard spares what is written inside its window, and this was written by some earlier commit. A collection that marked before this pointer landed sweeps it regardless. Re-checking afterwards catches that, and the pointer is removed rather than left dangling, since `mark` resolves every branch pointer and one dangling branch stops collection for the whole store. A caller that needs the state to survive a concurrent collection must pass the address as `extra-snapshots` to `gc!`; that is what it is for. THIS DROPS WHATEVER `branch` NAMED BEFORE. The superseded snapshot becomes unreachable and collectable — correct COW semantics, and the caller's business to hold if they still want it. It also does not disturb a live writer on the branch, because it cannot see one: reset a branch someone is writing and their next commit derives from the state they held, not this one. Returns the file map now at `branch`.
(reachable store)(reachable store extra-snapshots)One walk of the branch pointers: {:known :snapshots :addresses}.
THE SINGLE PLACE THE ROOT SET IS DERIVED, so every collector agrees on it.
mark treats extra-snapshots as hints and gc-cache! did not, because
gc-cache! reached the pointers through reachable-addresses instead —
which still resolved a vanished extra through read-snapshot and threw. The
documented usage was the failing one: an embedder whose held list lags by a
single entry got a gc! that worked and a gc-cache! that threw every time,
so the local pool was never reclaimed — exactly the unbounded-cache failure
gc-cache! exists to prevent, reported as a branch-pointer error.
Reading the pointers ONCE also matters: two walks let a commit land between them, leaving a snapshot whitelisted whose blobs had already been swept.
One walk of the branch pointers: `{:known :snapshots :addresses}`.
THE SINGLE PLACE THE ROOT SET IS DERIVED, so every collector agrees on it.
`mark` treats `extra-snapshots` as hints and `gc-cache!` did not, because
`gc-cache!` reached the pointers through `reachable-addresses` instead —
which still resolved a vanished extra through `read-snapshot` and threw. The
documented usage was the failing one: an embedder whose held list lags by a
single entry got a `gc!` that worked and a `gc-cache!` that threw every time,
so the local pool was never reclaimed — exactly the unbounded-cache failure
`gc-cache!` exists to prevent, reported as a branch-pointer error.
Reading the pointers ONCE also matters: two walks let a commit land between
them, leaving a snapshot whitelisted whose blobs had already been swept.(reachable-addresses store)(reachable-addresses store extra-snapshots)Every blob address named by a reachable snapshot — the GC root set.
Every blob address named by a reachable snapshot — the GC root set.
(reachable-snapshots store)(reachable-snapshots store extra)Snapshot addresses reachable from the branch pointers, plus extra.
extra is how an EXTERNAL HOLDER keeps an index state alive. A snapshot
address is immutable and safe to hand out, so a caller can store one and give
it back at collection time — which is exactly datahike's mark-from-key-map
contract. Without it, an index embedded in someone else's store is reachable
only from branch pointers, and a state they still reference but no branch
names is collected out from under them.
Snapshot addresses reachable from the branch pointers, plus `extra`. `extra` is how an EXTERNAL HOLDER keeps an index state alive. A snapshot address is immutable and safe to hand out, so a caller can store one and give it back at collection time — which is exactly datahike's `mark-from-key-map` contract. Without it, an index embedded in someone else's store is reachable only from branch pointers, and a state they still reference but no branch names is collected out from under them.
(read-manifest store branch)The branch's {lucene-filename -> address} map, or {} when it has none.
Two reads, not one: the pointer, then the snapshot it names. Against a remote store that is the cheaper shape rather than the more expensive one — the pointer is a few bytes and the snapshot is immutable, so a poller re-reads only the pointer and a cache keyed by address never needs invalidating.
The branch's `{lucene-filename -> address}` map, or `{}` when it has none.
Two reads, not one: the pointer, then the snapshot it names. Against a remote
store that is the cheaper shape rather than the more expensive one — the
pointer is a few bytes and the snapshot is immutable, so a poller re-reads
only the pointer and a cache keyed by address never needs invalidating.(read-snapshot store address)The commit at address: {:files {filename -> blob-address} :parents [...]}.
Throws when the address names nothing.
NOT {} on a miss. A pointer into a snapshot that does not exist is
corruption — a swept snapshot, an interrupted fork, an unreadable store — and
returning an empty map made every one of those look like an empty branch.
That is the worst possible reading: the mark then whitelists nothing for the
branch, so the next gc! deletes the blobs that would have made the damage
recoverable. Loud beats quiet here.
The commit at `address`: `{:files {filename -> blob-address} :parents [...]}`.
Throws when the address names nothing.
NOT `{}` on a miss. A pointer into a snapshot that does not exist is
corruption — a swept snapshot, an interrupted fork, an unreadable store — and
returning an empty map made every one of those look like an empty branch.
That is the worst possible reading: the mark then whitelists nothing for the
branch, so the next `gc!` deletes the blobs that would have made the damage
recoverable. Loud beats quiet here.(register-branch! store branch)Record branch in the registry, if it is not already there.
MUST HAPPEN BEFORE THE BRANCH'S FIRST MANIFEST WRITE, and that ordering is
the opposite of the values-then-pointer rule everywhere else here. Elsewhere
the pointer is written last because it makes values REACHABLE. The registry
is a GC ROOT: gc! whitelists a manifest only for a branch the registry
names, so a branch with a manifest the registry has forgotten has its
manifest and every blob it names swept. Registering first means a crash
leaves a registered branch with no manifest — harmless, read-manifest
returns {} — never the reverse.
Returns true when it wrote.
Record `branch` in the registry, if it is not already there.
MUST HAPPEN BEFORE THE BRANCH'S FIRST MANIFEST WRITE, and that ordering is
the opposite of the values-then-pointer rule everywhere else here. Elsewhere
the pointer is written last because it makes values REACHABLE. The registry
is a GC ROOT: `gc!` whitelists a manifest only for a branch the registry
names, so a branch with a manifest the registry has forgotten has its
manifest and every blob it names swept. Registering first means a crash
leaves a registered branch with no manifest — harmless, `read-manifest`
returns `{}` — never the reverse.
Returns true when it wrote.Lucene size knobs for an index whose segments are blobs in a remote store.
Lucene's defaults assume a local disk, where a segment is a file and a 5 GB one costs nothing to leave lying there. Against an object store the same segment is a blob written and read WHOLE, which changes what the defaults buy you:
:max-merged-segment-mb 256 (Lucene: 5120) The largest blob, and so the peak memory a commit costs — konserve's S3 backing assembles a blob in the heap to PUT it, roughly twice its size. It also has to stay clear of S3's 5 GB single-PUT ceiling, since that backing does not do multipart. 256 MB bounds the heap at ~0.5 GB and leaves an order of magnitude of headroom.
:ram-buffer-mb 32 (Lucene: 16) The other end of the distribution: this bounds segments created by a FLUSH, before any merge. Raised rather than lowered, because against a remote store the cost is per REQUEST — measured at ~5 objects per commit with a median blob of ~500 bytes — so fewer, larger flushes are cheaper than many small ones.
Pass to scriptum.core/create-index or open-branch. These are defaults to
start from, not tuned constants: the right cap depends on the heap you have
and how large the index gets.
Lucene size knobs for an index whose segments are blobs in a remote store.
Lucene's defaults assume a local disk, where a segment is a file and a 5 GB
one costs nothing to leave lying there. Against an object store the same
segment is a blob written and read WHOLE, which changes what the defaults
buy you:
:max-merged-segment-mb 256 (Lucene: 5120)
The largest blob, and so the peak memory a commit costs — konserve's S3
backing assembles a blob in the heap to PUT it, roughly twice its size.
It also has to stay clear of S3's 5 GB single-PUT ceiling, since that
backing does not do multipart. 256 MB bounds the heap at ~0.5 GB and
leaves an order of magnitude of headroom.
:ram-buffer-mb 32 (Lucene: 16)
The other end of the distribution: this bounds segments created by a
FLUSH, before any merge. Raised rather than lowered, because against a
remote store the cost is per REQUEST — measured at ~5 objects per commit
with a median blob of ~500 bytes — so fewer, larger flushes are cheaper
than many small ones.
Pass to `scriptum.core/create-index` or `open-branch`. These are defaults to
start from, not tuned constants: the right cap depends on the heap you have
and how large the index gets.(repair-branches! store)Rebuild the registry by scanning the keyspace for manifests.
The registry is authoritative, so nothing consults the keyspace on the read
path — which means drift, from a crash between registering and writing or
from a store assembled by other means, cannot repair itself. This is the way
back, and the expensive scan branches used to do on every call.
RUN IT QUIET. This scans manifests and then writes the registry
unconditionally, and it does NOT take the store lock the branch-pointer
operations use — the lock is defined below it and hoisting it is a change for
its own commit. So a delete-branch! running concurrently can be resurrected
as a registry entry for a branch that is gone, and a concurrent
register-branch! can be lost by the write. Neither loses data — reachable
discovers branches by scanning manifests, so an entry either way is corrected
there — but branches can be wrong until the next repair.
Returns the repaired set.
Rebuild the registry by scanning the keyspace for manifests. The registry is authoritative, so nothing consults the keyspace on the read path — which means drift, from a crash between registering and writing or from a store assembled by other means, cannot repair itself. This is the way back, and the expensive scan `branches` used to do on every call. RUN IT QUIET. This scans manifests and then writes the registry unconditionally, and it does NOT take the store lock the branch-pointer operations use — the lock is defined below it and hoisting it is a change for its own commit. So a `delete-branch!` running concurrently can be resurrected as a registry entry for a branch that is gone, and a concurrent `register-branch!` can be lost by the write. Neither loses data — `reachable` discovers branches by scanning manifests, so an entry either way is corrected there — but `branches` can be wrong until the next repair. Returns the repaired set.
Branch names that would collide with the cache's own layout.
A branch's view is cache/<branch>, and pool and snapshots are siblings
of it, so a branch with either name SHARES a directory with them. The damage
is not symmetric and pool is the bad one: opening that branch runs the
open-time reconcile, which deletes everything its manifest does not name —
the entire content-addressed pool, for every branch.
Branch names that would collide with the cache's own layout. A branch's view is `cache/<branch>`, and `pool` and `snapshots` are siblings of it, so a branch with either name SHARES a directory with them. The damage is not symmetric and `pool` is the bad one: opening that branch runs the open-time reconcile, which deletes everything its manifest does not name — the entire content-addressed pool, for every branch.
(snapshot-address files parents)The content address of a commit: files plus the parents it descends from.
A COMMIT HASH, NOT A TREE HASH, and the distinction is git's. Addressing the file map alone gives two commits with identical content the same address — so two branches that happen to produce the same segments would collide, and the second write would silently replace the first's parents. Covering the parents makes an address name a position in history rather than a set of bytes.
It also makes the merkle claim a real one. The values are blob addresses, themselves content hashes of segments, so a head address covers every segment in the index AND every ancestor: tamper with any byte at any point in the history and the head changes.
PARENTS, PLURAL, because a merge has two. merge-from! brings another
lineage in, and the codebase already models that one layer up — Lucene commit
user-data carries scriptum.parent-ids as a list and yggdrasil's
commit-info returns a set. A scalar here would have been the outlier, and
widening it later is precisely the migration this layout exists to avoid.
The cost is that two lineages holding identical content store the map twice. That map is metadata — measured at ~1KB across 200 commits — while the bytes live in blobs, which stay deduplicated because their addresses are unchanged.
The content address of a commit: `files` plus the `parents` it descends from. A COMMIT HASH, NOT A TREE HASH, and the distinction is git's. Addressing the file map alone gives two commits with identical content the same address — so two branches that happen to produce the same segments would collide, and the second write would silently replace the first's parents. Covering the parents makes an address name a position in history rather than a set of bytes. It also makes the merkle claim a real one. The values are blob addresses, themselves content hashes of segments, so a head address covers every segment in the index AND every ancestor: tamper with any byte at any point in the history and the head changes. PARENTS, PLURAL, because a merge has two. `merge-from!` brings another lineage in, and the codebase already models that one layer up — Lucene commit user-data carries `scriptum.parent-ids` as a list and yggdrasil's `commit-info` returns a set. A scalar here would have been the outlier, and widening it later is precisely the migration this layout exists to avoid. The cost is that two lineages holding identical content store the map twice. That map is metadata — measured at ~1KB across 200 commits — while the bytes live in blobs, which stay deduplicated because their addresses are unchanged.
(snapshot-directory store cache address)A READ-ONLY Directory over the index state at address.
This is what makes a snapshot address worth handing out. A caller who stored one — datahike's key-map, a reader pinned to a point in history — can open exactly the state it named, on any machine with the store, whatever the branch has done since.
Writes throw. The state is immutable by construction, and a Directory that accepted writes would have nowhere to put the result: there is no pointer to advance, only an address that already describes its own contents.
Materialization is lazy and shares the pool with every branch view, so opening a snapshot fetches only the files actually read and shares inodes with any branch holding the same blobs.
A READ-ONLY Directory over the index state at `address`. This is what makes a snapshot address worth handing out. A caller who stored one — datahike's key-map, a reader pinned to a point in history — can open exactly the state it named, on any machine with the store, whatever the branch has done since. Writes throw. The state is immutable by construction, and a Directory that accepted writes would have nowhere to put the result: there is no pointer to advance, only an address that already describes its own contents. Materialization is lazy and shares the pool with every branch view, so opening a snapshot fetches only the files actually read and shares inodes with any branch holding the same blobs.
(snapshot-files store address)Just the {filename -> blob-address} map at address.
Throws on a commit with no :files, rather than answering nil. Returning nil
here would undo exactly what read-snapshot throws for: mark would find no
blobs for the branch and, sweep! being allow-list, the next collection would
delete every segment it names.
Just the `{filename -> blob-address}` map at `address`.
Throws on a commit with no `:files`, rather than answering nil. Returning nil
here would undo exactly what `read-snapshot` throws for: `mark` would find no
blobs for the branch and, `sweep!` being allow-list, the next collection would
delete every segment it names.(snapshot-key address)An immutable commit — {:files {lucene-filename -> blob-address} :parents [...]}
— addressed by snapshot-address over both halves.
An immutable commit — `{:files {lucene-filename -> blob-address} :parents [...]}`
— addressed by `snapshot-address` over both halves.(snapshot-parents store address)The addresses this commit descends from: empty at the start of a lineage, one in ordinary history, two or more after a merge.
RECORDED BUT NOT YET WALKED. Nothing marks through them today, so a superseded commit is still collected exactly as before and chains stay short. They are here because retention needs them and adding them later would have forced a second layout change on published stores — the shape is settled now, and turning it on is a change to the mark rather than to the store.
When the mark does walk them, a parent that is not in the store must terminate the chain rather than raise: stores written before retention was switched on have their history collected already, and that is not corruption.
The addresses this commit descends from: empty at the start of a lineage, one in ordinary history, two or more after a merge. RECORDED BUT NOT YET WALKED. Nothing marks through them today, so a superseded commit is still collected exactly as before and chains stay short. They are here because retention needs them and adding them later would have forced a second layout change on published stores — the shape is settled now, and turning it on is a change to the mark rather than to the store. When the mark does walk them, a parent that is not in the store must terminate the chain rather than raise: stores written before retention was switched on have their history collected already, and that is not corruption.
Where read-only snapshot views live under the cache.
A sibling of pool rather than of the branch views, so gc-cache! can tell
them apart: it deletes a directory whose name is not a live branch, and a
snapshot view is by definition not one.
Where read-only snapshot views live under the cache. A sibling of `pool` rather than of the branch views, so `gc-cache!` can tell them apart: it deletes a directory whose name is not a live branch, and a snapshot view is by definition not one.
(store-id-for store)The guard id for store, or nil.
THE GUARD DEFAULTED TO OFF, AND THAT BRICKED BRANCHES ON THE ORDINARY PATH.
A nil id makes open-guard!/close-guard! no-ops and makes gc! take ts
instead of guard/cutoff, so a collection landing between the blob writes and
the pointer flip sweeps blobs the branch is about to name. The writer sees no
error; the branch head then names a blob that is gone and the branch cannot be
opened at all. Reproduced without instrumentation, bricking at commit 2.
IT IS KONSERVE'S ID OR NOTHING, DELIBERATELY. konserve.store/connect-store
requires a UUID :id and attaches the config to the store, so any store
connected that way answers here. konserve.filestore/connect-fs-store
bypasses that and carries no config, so it answers nil — do not use it.
Deriving a substitute from the store's path was tried and is worse than
refusing. The requirement is that every writer and collector on the SAME BYTES
agree, and a derived id is a different KIND of name from konserve's: one
component reaching a store through connect-store and another through
connect-fs-store would then hold a UUID and a path for one store — two ids,
which is the direction konserve.gc-guard calls out as deleting live data.
A single source of identity is the only way that cannot happen.
The guard id for `store`, or nil. THE GUARD DEFAULTED TO OFF, AND THAT BRICKED BRANCHES ON THE ORDINARY PATH. A nil id makes `open-guard!`/`close-guard!` no-ops and makes `gc!` take `ts` instead of `guard/cutoff`, so a collection landing between the blob writes and the pointer flip sweeps blobs the branch is about to name. The writer sees no error; the branch head then names a blob that is gone and the branch cannot be opened at all. Reproduced without instrumentation, bricking at commit 2. IT IS KONSERVE'S ID OR NOTHING, DELIBERATELY. `konserve.store/connect-store` requires a UUID `:id` and attaches the config to the store, so any store connected that way answers here. `konserve.filestore/connect-fs-store` bypasses that and carries no config, so it answers nil — do not use it. Deriving a substitute from the store's path was tried and is worse than refusing. The requirement is that every writer and collector on the SAME BYTES agree, and a derived id is a different KIND of name from konserve's: one component reaching a store through `connect-store` and another through `connect-fs-store` would then hold a UUID and a path for one store — two ids, which is the direction `konserve.gc-guard` calls out as deleting live data. A single source of identity is the only way that cannot happen.
(warm! store cache branch)(warm! store cache branch {:keys [only]})Materialize a branch's segments into the local cache, in parallel.
FOR A COLD MACHINE, where the store has everything and this machine has
nothing. Lucene's own warming hooks do not help here and it is worth being
precise about why: IndexInput.prefetch is called by the codecs, but this
Directory materializes a whole blob inside openInput before it ever returns
an IndexInput, so by the time Lucene hints the bytes are already local;
MMapDirectory.setPreload pages in files that are on disk; and
setMergedSegmentWarmer warms this writer's own merges. All three assume the
file is here. The fetch is ours to do.
IT HAS TO BE US ALSO BECAUSE LUCENE'S DEMAND IS SERIAL. StandardDirectoryReader
opens segment readers one at a time, so a cold query pays one round trip per
file in sequence — measured at 2.2 s for a 35-segment index against 60 ms
latency, against 348 ms for writing the same segments in parallel. There is no
concurrent demand to exploit; the only lever is fetching ahead of it.
EXPLICIT RATHER THAN AUTOMATIC. Materialization is lazy by design — a selective query should not pay for segments it never reads — and warming everything is the right call only when you know the machine is cold and about to serve. That is a caller's judgement, so it is a function rather than a policy.
:only is a predicate on the Lucene filename, for warming part of an index —
#(clojure.string/ends-with? % ".cfs"), say. Returns the number of files
materialized.
Safe alongside live readers and writers: materialization converges rather than
races (see link-into-view!), and a file already present costs one stat.
Materialize a branch's segments into the local cache, in parallel. FOR A COLD MACHINE, where the store has everything and this machine has nothing. Lucene's own warming hooks do not help here and it is worth being precise about why: `IndexInput.prefetch` is called by the codecs, but this Directory materializes a whole blob inside `openInput` before it ever returns an IndexInput, so by the time Lucene hints the bytes are already local; `MMapDirectory.setPreload` pages in files that are on disk; and `setMergedSegmentWarmer` warms this writer's own merges. All three assume the file is here. The fetch is ours to do. IT HAS TO BE US ALSO BECAUSE LUCENE'S DEMAND IS SERIAL. `StandardDirectoryReader` opens segment readers one at a time, so a cold query pays one round trip per file in sequence — measured at 2.2 s for a 35-segment index against 60 ms latency, against 348 ms for writing the same segments in parallel. There is no concurrent demand to exploit; the only lever is fetching ahead of it. EXPLICIT RATHER THAN AUTOMATIC. Materialization is lazy by design — a selective query should not pay for segments it never reads — and warming everything is the right call only when you know the machine is cold and about to serve. That is a caller's judgement, so it is a function rather than a policy. `:only` is a predicate on the Lucene filename, for warming part of an index — `#(clojure.string/ends-with? % ".cfs")`, say. Returns the number of files materialized. Safe alongside live readers and writers: materialization converges rather than races (see `link-into-view!`), and a file already present costs one stat.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |