scriptum.core — org.replikativ/scriptum 0.1.28

COW branching semantics on top of Apache Lucene.

Provides fast forking (~3-5ms), structural sharing of immutable segments, branch-isolated indexing/searching, snapshot retention, and explicit GC.

Key concepts:

• Writer: mutable handle to a branch (one per branch per JVM)
• Snapshot: immutable DirectoryReader at a specific commit point
• Branch: COW overlay sharing base segments with the trunk
• GC: explicit cleanup of old snapshots respecting branch references

->writer clj

(->writer sw-or-writer)

Extract the BranchIndexWriter from a ScriptumWriter or pass through a raw writer.

add-doc clj

(add-doc sw doc-map)

Add a document to the branch.

doc-map is a map of field-name -> value. Options per field: {:value v :type ... :store? bool}

Types: :text - Analyzed full-text (TextField) - default :string - Exact match (StringField) :int - Integer with range queries + sorting (IntField) :long - Long with range queries + sorting (LongField) :float - Float with range queries + sorting (FloatField) :double - Double with range queries + sorting (DoubleField) :stored-only - Store but don't index (StoredField) :vector - KNN float vector search (KnnFloatVectorField)

Auto-detection:

• java.time.Instant → :long (epoch millis)
• java.util.Date → :long (epoch millis)
• Vector of values → multi-valued field

Simple usage: (add-doc writer {:subject "Meeting notes" :from "[e-mail]" :date (Instant/now)})

Advanced usage: (add-doc writer {:subject {:value "Meeting" :type :text :store? true} :from {:value "[e-mail]" :type :string} :to {:value ["[e-mail]" "[e-mail]"] :type :string} :date {:value (Instant/now) :type :long :store? true} :size {:value 42000 :type :int :store? false} :headers {:value "{...}" :type :stored-only} :embedding {:value (float-array [...]) :type :vector :similarity :cosine}})

For fine-grained control, use Lucene classes directly: (let [doc (Document.)] (.add doc (TextField. "body" text Field$Store/NO)) (.add doc (StoredField. "body" text)) (.addDocument writer doc))

base-path clj

(base-path sw)

Returns the base path of the index.

bool-query clj

(bool-query clauses)

Build a BooleanQuery from clause specs.

Each clause is a vector of [query occur] where occur is one of: :must, :should, :must-not, :filter

Example: (bool-query [[(text-query "title" "clojure") :should] [(text-query "content" "clojure") :should] [{:term [:source "youtube"]} :filter]])

branch-name clj

(branch-name sw)

Returns the branch name.

branches clj

(branches sw)

Every branch of a store-backed index, from its manifests.

close! clj

(close! sw)

Close a branch writer and its resources.

commit! clj

(commit! sw)

(commit! sw message)

(commit! sw message metadata)

Commit changes on a branch. Stores timestamp in commit user-data.

Optional message is stored for history/log purposes. Optional metadata is a map of string keys to string values stored in commit user-data. Metadata keys must NOT use the "scriptum." prefix (reserved for internal use).

Returns a map with: :generation - the commit generation number :commit-id - Lucene's internal commit UUID :content-hash - content-addressable merkle root (only when :crypto-hash? enabled)

When :crypto-hash? is not enabled, returns just the generation number for backward compatibility.

Example with metadata (for secondary index sync): (commit! writer "Indexed tx" {"datahike.tx" "536870915"})

commit-available? clj

(commit-available? sw generation)

Check if a specific commit generation is still available (not GC'd).

create-index clj

(create-index path branch-name)

(create-index path
              branch-name
              {:keys [analyzer crypto-hash? max-merged-segment-mb
                      ram-buffer-mb]})

Create a new branched index at the given path.

On creation, discovers existing branches and protects their shared segments.

Options: :analyzer - the Lucene Analyzer to use (default: StandardAnalyzer) :crypto-hash? - enable merkle hashing for commits (default: false) :max-merged-segment-mb - cap on a merged segment, in MB (Lucene default: 5120) :ram-buffer-mb - flush buffer, in MB (Lucene default: 16)

THE TWO SIZE KNOBS ARE THE ONES THAT MATTER FOR A REMOTE STORE. Lucene's defaults are tuned for a local disk, where a segment is just a file and 5 GB costs nothing to leave lying there. Against an object store a segment is a blob written and read whole, so the merged-segment cap sets the peak memory a commit costs — konserve's S3 backing holds a blob in the heap to PUT it — and it has to stay clear of S3's 5 GB single-PUT limit. A few hundred MB is a reasonable cap there; scriptum.konserve/remote-tuning carries defaults.

The flush buffer sets the other end of the distribution: it bounds segments created by a flush, before any merge, and so governs how small the small objects are.

Returns a ScriptumWriter wrapping BranchIndexWriter + metadata index.

delete-docs clj

(delete-docs sw field value)

Delete documents matching the given term field and value.

discover-branches clj

(discover-branches path)

Discover all branch names at the given path.

Returns a set of branch name strings.

find-generation clj

(find-generation sw key value)

(find-generation sw key value mode)

Find the commit generation matching a custom metadata key/value.

mode can be: :exact - exact match (default) :floor - latest commit whose metadata value <= target (for monotonic values like tx IDs)

Returns nil if no match, or a map with :generation (and :indexed-value for :floor mode).

Example: (find-generation writer "datahike/tx" "536870915") (find-generation writer "datahike/tx" "536870915" :floor)

flush! clj

(flush! sw)

Flush pending changes without committing (no durability, but NRT visible).

fork clj

(fork sw new-branch-name)

Fork the index into a new branch. Returns the new branch writer.

The new branch shares all existing segments with the parent. Cost: ~3-5ms (flush buffer + copy manifest).

gc! clj

(gc! sw before)

Garbage collect old commit points and unreferenced segment files.

Only callable on the main branch writer. Scans all branches to determine which files are still needed before removing anything.

IT RECLAIMS NOTHING WHILE A BRANCH STILL SHARES FILES WITH MAIN, which after an ordinary fork is always. Protection is per COMMIT POINT: one is spared if it references any file some branch references, and a fresh fork shares every base segment by construction. Measured over 6 commits:

no branch 6 removed one fork, untouched 0 removed (and the call adds a commit point) one fork, closed 0 removed fork force-merged onto its own segment 7 removed empty branch directory 6 removed

So the condition is narrower than a branch exists: a branch that has merged away its inherited segments stops protecting them, and an empty directory protects nothing. But the common case — fork and keep working — does pin every commit point on main indefinitely, and a call that reclaims nothing still adds one, so history grows.

The conservatism is in the safe direction: nothing is deleted that a branch might need. Fixing it means protecting FILES rather than commit points, which Lucene's deletion-policy interface cannot express directly.

The store-backed model does not have the SHARING problem — reachability is computed across every branch's manifest, so a shared segment is protected by being named rather than by freezing the commit point that names it. It still needs retain! to drop commit points before anything becomes unreachable; neither model collects history you have not asked it to drop.

before: java.time.Instant — delete commits older than this Returns the number of commit points removed.

list-snapshots clj

(list-snapshots sw)

List all available snapshots (commit points) for this branch.

Returns a vector of maps with :generation, :snapshot-id, :timestamp, :message, :branch, :segment-count, :parent-ids, and :custom-metadata.

:custom-metadata is a map of any non-scriptum keys stored in commit user-data.

main-branch? clj

(main-branch? sw)

Returns true if this is the main (trunk) branch.

max-doc clj

(max-doc sw)

Returns the total number of documents (including deletions).

merge-from! clj

(merge-from! target source)

Merge segments from a source branch into this branch.

Uses reader-based addIndexes to avoid lock conflicts with source writer.

multi-field-query clj

(multi-field-query fields text)

(multi-field-query fields text analyzer)

Parse a text query string across multiple fields.

Each token is searched across all given fields with SHOULD semantics (match in any field counts).

Args: fields - seq of field names (strings or keywords) text - query string analyzer - Lucene Analyzer (optional, defaults to StandardAnalyzer)

Example: (multi-field-query ["title" "content"] "clojure reactive") ;; Matches docs where title OR content contains 'clojure reactive'

num-docs clj

(num-docs sw)

Returns the number of documents in this branch (excluding deletions).

open-branch clj

(open-branch path branch-name)

(open-branch path
             branch-name
             {:keys [analyzer metadata-index max-merged-segment-mb
                     ram-buffer-mb]})

Open an existing branch writer (for out-of-process branch access).

Opens a BranchedDirectory with the base as read-only and overlay for writes.

Options: :analyzer - the Lucene Analyzer to use (default: StandardAnalyzer) :metadata-index - shared metadata index (default: creates new one) :max-merged-segment-mb - cap on a merged segment, in MB (Lucene default: 5120) :ram-buffer-mb - flush buffer, in MB (Lucene default: 16)

THE TWO SIZE KNOBS ARE THE ONES THAT MATTER FOR A REMOTE STORE. Lucene's defaults are tuned for a local disk, where a segment is just a file and 5 GB costs nothing to leave lying there. Against an object store a segment is a blob written and read whole, so the merged-segment cap sets the peak memory a commit costs — konserve's S3 backing holds a blob in the heap to PUT it — and it has to stay clear of S3's 5 GB single-PUT limit. A few hundred MB is a reasonable cap there; scriptum.konserve/remote-tuning carries defaults.

The flush buffer sets the other end of the distribution: it bounds segments created by a flush, before any merge, and so governs how small the small objects are.

open-reader-at clj

(open-reader-at sw generation)

Open a reader at a specific commit generation (time-travel).

The caller is responsible for closing the reader. Throws if the generation has been GC'd.

open-store-index clj

(open-store-index store cache branch)

(open-store-index store
                  cache
                  branch
                  {:keys [analyzer metadata-index store-id max-merged-segment-mb
                          ram-buffer-mb]})

Open branch of a konserve-backed index, materializing through cache.

The store is the source of truth; cache is a derived local directory that may be deleted at any time — see scriptum.konserve. Lucene still mmaps local files, so a cache is required even when the store is remote; what the store buys is that it is the only thing that must be durable.

Takes a CONNECTED store. A secondary index that must reconnect from a serialized key-map (datahike's -sec-restore) connects it itself and passes the store in — that belongs in the adapter, which owns the config, not here.

Options: :analyzer - the Lucene Analyzer (default: StandardAnalyzer) :metadata-index - shared metadata index (default: none) :store-id - id for konserve.gc-guard, so a collection cannot sweep this index's in-flight segment writes. Defaults to the store's own id, which is what keeps two components on one store from disagreeing about its name. :max-merged-segment-mb / :ram-buffer-mb - see create-index. Against a remote store start from scriptum.konserve/remote-tuning.

Returns a ScriptumWriter. Document operations, search, commit and readers behave exactly as for a directory-backed index, and fork and branches answer from the manifests instead of the filesystem. COLLECTION IS DIFFERENT: scriptum.core/gc! throws here and scriptum.konserve/gc! is the one to call, because a store-backed index collects by reachability and has to read the gc-guard's cutoff before walking, which the directory-backed collector does not do.

open-store-index-at clj

(open-store-index-at store cache branch address)

(open-store-index-at store cache branch address opts)

Open branch as a WRITABLE index at the state named by address.

Points the branch at address and opens it — the restore half of snapshot-address. Without it a holder could read a snapshot (scriptum.konserve/snapshot-directory) but never write from one, so restoring a secondary index to a specific state was impossible and opening the branch silently gave whatever it had moved on to instead.

THIS MOVES THE BRANCH. Whatever it named before becomes unreachable and collectable; hold that address yourself if you still want it. Do not call it on a branch another writer has open — see scriptum.konserve/point-branch-at!.

Takes the same options as open-store-index.

retain! clj

(retain! sw {:keys [before commit-ids]})

Drop old commit points from a store-backed index, bounding its growth.

THE THING THAT MAKES A STORE-BACKED INDEX FINITE. Nothing else prunes it: every commit point is kept, so the branch's file map is cumulative — 30 commits of 30 documents were measured naming 130 files, 30 of them commit points — and all of it is legitimately reachable, so scriptum.konserve/gc! correctly reclaims nothing. Dropping a commit point removes its files from the manifest, and the collector can then take the blobs no other branch names.

Two ways to say what goes, because two callers ask different questions:

:before — an Instant; drop commit points committed before it. This is yggdrasil's :remove-before, and the timestamp compared is the real commit time from user-data. :commit-ids — drop exactly these snapshot-ids. yggdrasil's coordinator computes reachability itself, from every system's gc-roots and the commit graph, and hands each adapter its own candidates; a cutoff cannot express that, since an unreachable commit may be newer than a reachable one elsewhere.

Issues a commit WHEN IT WILL ACTUALLY DROP SOMETHING, because onCommit is the only place Lucene lets a deletion policy act — and because committing is not free here, since the commit itself becomes a commit point. A sweep that matches nothing returns 0 without touching the index; doing otherwise made the collector grow the index on every cycle under yggdrasil, whose candidates come from a registry that never names scriptum's own bookkeeping commits.

THE BRANCH HEAD IS NEVER DROPPED, whatever the cutoff says, so a caller who passes :before (now) does not lose the commit gc-roots just reported.

THE SHRINK IS PUBLISHED AT THE NEXT COMMIT, not this one: Lucene removes a dropped commit point's files during the checkpoint that follows the flip, so the manifest for the retain commit is already written by then. retain! reports what it dropped; the manifest reflects it once you commit again.

IT ALSO COMMITS WHATEVER IS BUFFERED, because IndexWriter.commit cannot do otherwise. Collection should not be what makes a caller's in-flight writes durable — commit first if that distinction matters to you.

READING A DROPPED COMMIT BY GENERATION STOPS WORKING — that is the trade. Its state is still reachable by snapshot address, which is what scriptum.konserve/snapshot-directory opens and what yggdrasil's as-of maps onto.

HOLDING AN ADDRESS PINS NOTHING. snapshot-address hands you a value; it registers no claim on it, and once no branch names that state gc! collects it. To keep one you must pass it as extra-snapshots on EVERY collection — to scriptum.konserve/gc! for the store and to gc-cache! for the local cache, which take it separately.

Returns the number of commit points dropped, or nil for a directory-backed index, which must use gc! — there a commit point holds real files another branch may share.

search clj

(search sw query)

(search sw query {:keys [limit fields reader] :or {limit 10}})

Search a branch. Returns a vector of maps with :doc-id, :score, and field values.

query can be:

• A Lucene Query object
• A map {:term [field value]} for a term query
• A string (matches all documents containing this term in any field)

Options: :limit - max results (default 10) :fields - fields to retrieve (default: all stored fields) :reader - search THIS reader instead of opening one (see below)

BY DEFAULT THIS OPENS A FRESH NRT READER, so it reflects the writer's state including uncommitted changes — add a document and it is findable before any commit. That is the semantics a git-like writer wants and it is not free: DirectoryReader.open(writer) flushes every in-memory buffer, so a loop that alternates writing and searching materializes a segment PER SEARCH. Measured at 5.08 ms per write-then-search cycle against 0.159 ms reusing a reader — 32x, and the cost is segment churn rather than reader construction, which is cheap (0.016 ms at one segment, 0.121 ms at 54).

So pass :reader when searching repeatedly without writing, or when writing and searching in a loop. snapshot, with-snapshot and open-reader-at hand out exactly the right object, which is also what makes this compose with time travel. Whoever opens the reader closes it; scriptum owns no lifecycle here, deliberately.

A held reader is a POINT IN TIME. It will not show later writes, and — more sharply — it will still return documents deleted since, so a caller filtering on identity must expect rows it has already removed. Reopen or take a fresh snapshot to move forward.

Scriptum caches no searcher of its own, and that is a decision rather than an omission: a cached NRT searcher refreshed on commit! was measured to break read-your-own-writes, resurrect deleted documents between refreshes, and miss merge-from! entirely — 5 documents against 13 — because that path commits inside the Java layer without passing through commit!. The realistic gain was 1.05-3.5x on a mixed query load, which is a poor price for those.

snapshot clj

(snapshot sw)

Take an immutable snapshot (DirectoryReader) of the current branch. The caller is responsible for closing the reader.

snapshot-address clj

(snapshot-address sw)

The immutable address of this branch's current index state, or nil.

THE VALUE A CALLER HOLDS TO COME BACK TO THIS EXACT STATE. A branch name is a mutable cell and says nothing about which commit it is on; this is content- addressed and cannot change under the holder. It is what a secondary-index key-map should carry — datahike's already carries :commit-id for proximum and :dataset-commit-id for stratum, and scriptum was the outlier naming a branch.

It is also a merkle root over the whole history — ContentHash/hashMap over the file map AND the parents, whose values are themselves content hashes of segments — so it doubles as a content hash without :crypto-hash? being on.

Reflects the last COMMIT, since the branch pointer moves at commit time — buffered writes are not in it. Store-backed indices only; nil otherwise.

store-backed? clj

(store-backed? sw)

Is this writer backed by a konserve store rather than a directory tree?

text-query clj

(text-query field text)

(text-query field text analyzer)

Parse a text query string against a single field using the given analyzer.

Uses Lucene's QueryParser to handle operators (+, -, AND, OR, NOT), phrases ("quoted text"), wildcards (*), and fuzzy matching (~).

Args: field - field name to search (string or keyword) text - query string analyzer - Lucene Analyzer (optional, defaults to StandardAnalyzer)

update-doc clj

(update-doc sw field value doc-map)

Update a document identified by the given term.

Replaces the document matching (field, value) with the new doc-map. doc-map uses the same format as add-doc (supports all field types, multi-valued fields, auto-detection).

verify-commit clj

(verify-commit sw)

(verify-commit sw {:keys [generation] :or {generation -1}})

Verify the cryptographic integrity of a commit by recomputing its merkle hash.

Requires that the index was created with :crypto-hash? true.

Options: :generation - commit generation to verify (default: -1 for current HEAD)

Returns a map with: :valid? - boolean indicating if verification passed :commit-id - the commit UUID that was verified :errors - vector of error messages (empty if valid)

Example: (verify-commit writer) ; verify current commit (verify-commit writer {:generation 5}) ; verify specific generation

warm! clj

(warm! sw)

(warm! sw opts)

Materialize this branch's segments into the local cache, in parallel.

FOR A COLD MACHINE — a fresh container, a thawed Lambda, a cache that was wiped. The store has everything and this machine has nothing, and Lucene will otherwise fetch one file per round trip in sequence, because StandardDirectoryReader opens segment readers serially. Measured on a 35-segment index at 60 ms latency: 2.2 s lazily against 275 ms warmed.

Explicit rather than automatic: materialization is lazy by design, since a selective query should not pay for segments it never reads. Warming is worth it when you know the machine is cold and about to serve.

Options: :only, a predicate on the Lucene filename. Returns the number of files materialized. Store-backed indices only; nil otherwise.

with-snapshot clj

(with-snapshot sw f)

Execute f with an immutable snapshot reader. Reader is closed after.