v0.1.0 shipped the feature-complete core (storage engine, gRPC+REST, 7 client SDKs, web UI, durability modes, metrics, TLS). This document plans the next arc: making the durability promise real, making queries scale, and rounding out the feature set.
It is derived from the codebase review of 2026-06-29. Each item lists the
problem (with file:line evidence), the approach, proto/API impact, the
files touched, tests, and an acceptance bar. Items are grouped into
release milestones and sized S (≤½ day), M (1–2 days), L (3+ days).
Workflow: one PR per task. Small correctness fixes are batched into a single hardening PR; each large feature is its own branch (or warden agent in an isolated worktree). CI must be green before merge; every feature updates
docs/,README.md, and ticks its box here.
| Milestone | Theme | Items | Risk | Headline |
|---|---|---|---|---|
| v0.2.0 | Durability & correctness hardening | D1–D6 | Low | Back the durability promise; no silent data/edge bugs |
| v0.3.0 | Query at scale | Q1–Q4 | Medium | Stop materializing whole collections; range indexes |
| v0.4.0 | Feature breadth | F1–F3 | Medium | TTL, backup/snapshot, on-demand compaction |
| v0.5.0 | Auth & multi-tenancy | A1 | Medium | Multiple scoped, rotatable API keys |
Small, high-trust-impact fixes. Ship D1–D2 + D5 first as one PR (they directly
back the --sync feature), then D3/D4/D6 as follow-ups.
segment.go fsyncs file contents (Sync, Seal) but never
the parent directory. A new seg_NNNNNN.ndjson created in rotateSegment
(collection.go:531) or openActiveSegment (segment.go:30) is not durable
on crash even under SyncModeAlways. Same for the atomic-rename of
index.json / sidx_*.json (index.go:98, secondary_index.go:182) —
rename is atomic for visibility, not durability, without a dir fsync.fsyncDir(path string) error helper (open dir, Sync,
close). Call it after: first active-segment create, every rotation, and every
tmp→final rename of index/sidx/meta files. Gate the per-rotation dir-fsync on
SyncMode != none to preserve fast-mode throughput.engine/segment.go, collection.go, index.go,
secondary_index.go, new engine/fsync.go.durability_test.go — extend the reopen matrix; add a test that
asserts dir-fsync is invoked per sync mode (inject a hook/counter).always, no acknowledged write or its containing
segment file can be lost across a simulated crash (kill + reopen).meta.json writes — SpersistMeta uses in-place os.WriteFile (meta.go:38) — a
torn write corrupts it — and it runs on every insert (collection.go:314),
adding a file write to the hot path even in SyncModeNone.persistMeta to tmp+rename like the indexes; (b) stop
writing it per-insert — persist on rotation, on Close, and on a coarse timer
(or every N writes). The id counter is already recoverable by scan, so this is
pure optimization + safety, not a correctness regression.engine/meta.go, collection.go.collection_test.go — reopen after N inserts with no clean close
still recovers the correct id counter (falls back to scan); meta written
atomically (no partial file observable).none mode improves measurably
(make bench); id counter never regresses across crash/reopen.store.Decode
returns wrong data if the corrupted bytes still parse as JSON.crc field to the NDJSON entry (CRC32C of the
canonical data), written by store.Encode and verified by store.Decode
when present. Keep it backward-compatible: absent crc = legacy line, no
verification. Verification failures surface as a typed ErrCorruptEntry.docs/architecture.md.store/ndjson.go, engine/segment.go
(ScanAll/ReadAt error wrapping), docs/architecture.md.ndjson_test.go — round-trip with/without crc; flip a byte →
ErrCorruptEntry. segment_recovery_fuzz_test.go — extend fuzz corpus.emit drops events when a subscriber's 64-buffer is full
(collection.go:524, default: case) with no notification — consumers
silently miss writes.WatchEvent{Op: OpOverflow} (new op) once the
channel drains, so the client knows to resync. Make buffer size configurable.OVERFLOW to the WatchEvent op enum in proto/scriva.proto;
regenerate stubs; map in server/grpc.go + server/watch_rest.go.proto/scriva.proto, engine/collection.go,
server/grpc.go, server/watch_rest.go, web UI feed handling.collection_test.go — slow subscriber receives an overflow sentinel
rather than silently missing events.Update, Delete, and CommitTx discard the rotation result
(_ = c.rotateSegment(), e.g. collection.go:347, 655); only Insert
checks it. A failed rotation (e.g. disk full) is silently ignored on three of
four write paths.Insert's pattern at
collection.go:309). The write itself already succeeded, so wrap as a
non-fatal warning return or a logged error — decide one consistent policy and
apply it on all four paths.engine/collection.go.collection_test.go — inject a rotation failure; assert it is
surfaced (not swallowed) on every write path.TxManager (txmanager.go) never evicts transactions. A client
that calls BeginTx and disconnects leaks the *Tx, its staged ops, and its
reserved id forever.Tx with createdAt/lastUsed; run a sweeper
goroutine that rolls back + removes transactions idle beyond a configurable
TTL (default 5m). Expose --tx-timeout (Config + YAML + flag, per the
"Adding a new server flag" checklist in CLAUDE.md).engine/txmanager.go, server/config.go,
cmd/scriva/main.go.txmanager_test.go — idle tx is reaped after TTL; active tx is
not; reaped tx commit fails cleanly.The flagship arc. Today every non-indexed query materializes the entire
collection in RAM (Scan slow path, collection.go:425) and limit/offset/
order_by are applied after the fact in the gRPC handler (grpc.go:151-178),
so Find ... limit 10 still reads the whole collection.
limit; the streaming
RPC streams results that were already fully buffered.map[uint64]Entry.limit/offset into the engine; stop scanning once offset+limit
matches are produced (when no order_by, or when ordering by an indexed
field).ScanResults through a channel to Find so the server emits as it
reads rather than buffering. Respect ctx cancellation (see Q4).order_by is present and unindexed, fall back to bounded buffering
(top-K heap of size offset+limit) rather than full sort of all rows.limit/offset/order_by already exist);
semantics tightened. Document ordering guarantees.engine/collection.go (new ScanStream), server/grpc.go
(consume the stream), docs/architecture.md.collection_test.go + grpc_integration_test.go — large-collection
bench shows limit 10 reads/holds O(limit), not O(n); top-K ordering correct.offset+limit, not collection size (verified by a bench/metric).order_by — S/Mfmt.Sprintf("%v") string compare with only a
float64 fast path (grpc.go:153); ints-as-json.Number and mixed types sort
lexically, and there's no descending option.query/filter.go
(it already orders float/int/string/json.Number for gt/lt). Add a sort
direction (order_dir / -field convention). Centralize comparison so filter
and sort share one code path.string order_dir = 6; (or reuse a --prefix) to
FindRequest; regenerate.proto/scriva.proto, query/filter.go (export a
Compare), server/grpc.go, CLI find flag, clients/docs.filter_test.go + integration — numeric vs string ordering, desc.order_by age sorts 2 < 10; desc reverses correctly.OpEq (collection.go:411);
gt/gte/lt/lte always full-scan despite being supported filters.Scan picks the index
for range predicates and AND-combines candidate id sets. Persist with the same
tmp+rename+checksum pattern; rebuild after compaction.EnsureIndex with an index-type hint
(hash | ordered); default ordered so it serves both eq and range.engine/secondary_index.go (or new ordered_index.go),
collection.go (planner), proto/scriva.proto, server/grpc.go, CLI, docs.secondary_index_test.go — range lookups match full-scan results;
survives update/delete/compaction; persistence round-trips.gt/lt query on an indexed field reads O(matches), not
O(collection), and returns identical results to the scan path.ctx into the engine, so a client
cancelling a long Find/Scan doesn't stop server-side work.ctx params to ScanStream/long reads; check
ctx.Err() between segments/batches and abort. Pairs naturally with Q1.engine/collection.go, server/grpc.go.store.Entry.ExpiresAt (Unix-nano, folded into the CRC,
omitted when unset — backward compatible) and a mirrored IndexEntry.ExpiresAt
so reads drop expired records without a disk hit. Engine API:
InsertWithExpiry / UpdateWithExpiry (explicit deadline) and
CollectionConfig.DefaultTTL (now+TTL on inserts lacking one). Updates keep a
record's deadline (sticky) unless UpdateWithExpiry overrides it. Server
--default-ttl flag (Config + YAML + flag). Reaper in compactLoop;
resolveEntries drops expired entries during compaction.expires_at / ttl_seconds on the
Insert/Update RPCs and per-collection default TTL in CreateCollection, plus
the 7 language clients. Kept out of this PR to keep it engine-first and
reviewable, matching the KEY-1 precedent; TTL is fully usable via the embedded
engine and the server-wide --default-ttl.store/ndjson.go, engine/index.go, engine/collection.go,
engine/ttl.go, engine/compactor.go, engine/keys.go, engine/scan.go,
server/config.go, cmd/scriva/main.go, docs.engine/ttl_test.go — hidden after deadline, visible before,
default-TTL applied, sticky across update, override, reaped, reclaimed by
compaction, survives reopen, excluded from secondary-index lookups;
store round-trip + omitempty.scriva-cli backup <dest> + a streaming Snapshot RPC that
produces a consistent gzip tarball of the data dir. Restore is just untar into
--data.DB.SnapshotTo(io.Writer) writes a gzip-compressed tar of every
collection's files. Consistency: the DB registry is held read-locked for the
whole archive (no create/drop/reopen mid-snapshot) and each collection's files
are copied under its own read lock (no write, rotation, or compaction during
the copy). The primary index.json is excluded — it stores absolute
segment paths and a self-only checksum, so the restored collection rebuilds it
from segments on open; secondary indexes (path-independent value→id maps) are
refreshed and included. Streaming Snapshot RPC (gRPC-only; binary streaming
does not map cleanly onto REST) sends 64 KiB gzip chunks. scriva-cli backup <dest> writes the stream to a .tar.gz and prints the restore command; REPL
backup verb added.Snapshot RPC.engine/snapshot.go, engine/db.go (via SnapshotTo),
proto/scriva.proto, server/grpc.go, cmd/scriva-cli/commands.go,
cmd/scriva-cli/main.go, cmd/scriva-cli/repl.go, docs.engine/snapshot_test.go — round-trip (multi-segment, update +
delete + secondary index) restores to identical query results, empty-DB
archive; server/grpc_integration_test.go TestIntegration_Snapshot — stream
the RPC, extract, reopen, verify record count.Compact(collection) RPC that runs a synchronous,
forced compaction pass and returns only after it completes;
scriva-cli compact <collection>.Collection.CompactNow() runs a forced pass (bypassing the
dirty-ratio gate) and DB.Compact(name) wraps it. Background and on-demand
passes serialize via a new compactMu so they never race the sealed-segment
swap; compact(force bool) skips the dirty gate when forced. New Compact
RPC (POST /v1/{collection}/compact), server/grpc.go handler, and
scriva-cli compact <collection> (plus a REPL compact verb). A closed
collection refuses to compact.Compact RPC.proto/scriva.proto, engine/compactor.go (synchronous forced
entry + serialization), engine/collection.go, engine/db.go,
server/grpc.go, cmd/scriva-cli/commands.go, cmd/scriva-cli/main.go,
cmd/scriva-cli/repl.go, docs.engine/compact_ondemand_test.go — forces a merge below the dirty
threshold and reduces segment count, refuses on a closed collection;
server/grpc_integration_test.go TestIntegration_Compact — RPC round-trip,
records survive, unknown collection is NotFound.compact returns after a full compaction pass completes.internal/auth/apikey.go).keys: list.internal/auth/, server/config.go, cmd/scriva/main.go, docs.auth unit tests — scope enforcement, unknown key rejected,
reload picks up new keys; constant-time compare preserved.v0.2.0 PR-A: D1 + D2 + D5 (durability hardening — do first)
PR-B: D6 (tx GC) + new --tx-timeout flag
PR-C: D4 (watch overflow) + proto enum
PR-D: D3 (record checksums)
v0.3.0 Agent-1: Q1 + Q4 (streaming read path — flagship)
PR-E: Q2 (typed order_by)
Agent-2: Q3 (range indexes)
v0.4.0 Agent-3: F1 (TTL)
PR-F: F2 (backup) , PR-G: F3 (compact)
v0.5.0 Agent-4: A1 (scoped keys)
Each large item (Q1, Q3, F1, A1) is a candidate warden development agent in its
own worktree; the small items ship as direct PRs. Every merged item ticks its
box above and updates ROADMAP.md, docs/, and README.md per CLAUDE.md
conventions.
v0.2.0 — Hardening
meta.json--tx-timeoutv0.3.0 — Query at scale
order_byv0.4.0 — Features
ttl_seconds + per-collection default_ttl_seconds surfaced on the RPCs & CLI)Snapshot RPC + scriva-cli backup)v0.5.0 — Auth
keys: list + SIGHUP reload)Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |