Scope note: this is a use-case roadmap, separate from the release roadmap in
ROADMAP.md/docs/roadmap-v0.2.md. It tracks only the features ScrivaDB needs so that warden can adopt it as an embedded, in-process store — no sidecar server, no gRPC, no API key. Items here are sized S (≤½ day), M (1–2 days), L (3+ days) and, where they overlap the release roadmap, cross-reference it instead of duplicating.
warden is a single-user, single-machine, single static binary daemon. Its
current store is embedded in the daemon process (an RWMutex-guarded
per-file JSON store; the daemon is the only holder). The only architecture that
preserves warden's single-binary install and zero-sidecar footprint is to
compile ScrivaDB's engine into warden and call it in-process:
warden daemon
└── import github.com/srjn45/scriva/engine ← in-process, no network
└── engine.DB ──► data/warden/<collection>/seg_*.ndjson
So the server, REST gateway, gRPC, and API-key layers are out of scope for this use case — warden never talks to a socket. Everything below is about making the engine a first-class embeddable library.
The non-negotiable blocker is that the engine currently lives in
internal/engine, which Go forbids other modules from importing. Until EMB-1
lands, warden can only run ScrivaDB as a separate server — which we've ruled out.
Today warden co-locates or single-files several things that should be their own collections. ScrivaDB's native model (one directory per collection, one record per line, append-only) gives us that split for free — modeling each concern as a collection is what removes the whole-file-rewrite cost.
| warden store (today) | Today's shape | Problem | ScrivaDB collection (target) | Key |
|---|---|---|---|---|
internal/store sessions | one JSON file / session, events embedded as a growing slice | every status tick rewrites the whole growing file (O(n) write-amp) | sessions | session id (string) |
| — (events) | embedded Events []Event in the session file | see above | events (append-only) | session id (secondary index) + ts |
internal/mailbox | one JSON file / recipient, full rewrite on every append | write-amp per message; inbox grows unbounded | messages (append-only) | recipient id (secondary index) |
internal/ctxstore | single context.json, whole map rewritten on every Set/Append, CAS | single-file, doesn't scale; rewrite on each write | context | context key (string), version for CAS |
internal/spend | ledger | ok-ish | spend | — |
internal/savings | NDJSON ledger (already append-only ✅) | already good — the pattern we're generalizing | savings | — |
internal/pipeline | store | — | pipelines | pipeline id |
internal/schedule | single schedules.json (few, low-freq) | fine as-is; migrate for uniformity | schedules | schedule id |
internal/snapshot | store | — | snapshots | snapshot id |
Net effect for warden: sessions becomes a thin, bounded record; the
unbounded, append-heavy data (events, messages) moves to append-only
collections where a write is a single line, not a full-file rewrite; and the
single-file context.json becomes a keyed collection with real CAS. This is the
concrete answer to "put session data, messages, and context into separate
files."
| Milestone | Theme | Items | Why it's needed for warden |
|---|---|---|---|
| EMB | Embeddable engine | EMB-1 … EMB-4 | Make engine importable, ergonomic, and dependency-light |
| KEY | Keys, uniqueness & CAS | KEY-1 … KEY-4 | warden uses string ids, unique names, and compare-and-swap writes |
| QRY | Query fit | QRY-1 … QRY-3 | List newest-first, lookups, counts — mostly cross-refs release roadmap |
| OPS | Embedded operations | OPS-1 … OPS-5 | durability default, in-process Watch, migration, backup |
internal/engine cannot be imported from the warden module
(internal/ is module-private). This blocks the entire integration.internal/engine → a public path (engine/, or pkg/engine),
and repoint the server (server/grpc.go) and CLI at the public package so there
is exactly one engine. Keep internal/store, internal/query either public too
(they're on the public API surface: store.Entry, query.Filter) or re-export
the needed types from engine. Nothing about behavior changes — this is a
visibility + package-move refactor guarded by the existing test suite.engine.Open, engine.DB, engine.Collection
(Insert/Update/Delete/FindByID/Scan/Watch), engine.CollectionConfig,
engine.SyncMode, query.Filter constructors, store.Entry.internal/engine/* → engine/*; update imports in server/,
cmd/; internal/query, internal/store visibility.import "github.com/srjn45/scriva/engine"
and open a DB, with no internal/ reference.engine.Open(dataDir, CollectionConfig) opens with a single
config; warden wants a DB that hosts several collections (sessions, events,
messages, context, …) with per-collection config, and a dead-simple
"open a store rooted at this dir" call.scriva.Open(dir, opts...)
returning a handle with Collection(name, cfg) / MustCollection), defaulting
sync mode, segment size, and compaction cadence sensibly for a local daemon.
This is sugar over the existing DB — no engine changes.scriva package (or engine.OpenDB helper), docs.engine transitively drags in gRPC, grpc-gateway,
protobuf, cobra, and Prometheus, warden's binary and go.mod bloat for code it
never runs. The embed path must pull only the engine's real deps
(std lib + minimal).engine/query/store do not import server/, pb/,
auth/, metrics/, or cobra. Metrics stays behind the existing
OnCompaction hook (an interface warden can ignore). Verify with
go mod graph / go list -deps that importing engine brings in no
grpc/protobuf/prometheus. Consider a go.mod tool/build-tag split only if a
stray dep can't be untangled.make deps-check) asserting the
embed surface's dependency set.embeddemo module that imports only engine and builds; CI
fails if the dep set grows.go list -deps ./engine contains no grpc/protobuf/prometheus/cobra.github.com/srjn45/scriva as a library dependency;
it needs a stable, versioned API surface, not "whatever main is."v0.x releases
that warden can go get, and note any breaking change policy. (Distribution for
this use case is go get, not brew/apt/GHCR — those stay for the standalone
server.)README.md (embedding section), docs/embedding.md (new), CHANGELOG.go get github.com/srjn45/scriva/engine@vX.Y.Z works and the
documented API matches.uint64 auto-ids
(Insert(data) (uint64, ...), map[uint64]IndexEntry). warden's identifiers
are strings (session ids, agent names, context keys, recipient ids). Storing
the warden id only as a data field forces an index lookup on every access and
loses primary-key uniqueness.map[string]IndexEntry with the
numeric path as a special case; (b) add InsertWithKey(key string, data) +
FindByKey(key) backed by a mandatory unique index on a reserved _key field.
Must round-trip through segments, compaction, and index rebuild.engine/index.go, engine/collection.go, store/ndjson.go (entry
carries the key), docs.FindByKey("sess-abc123") is O(1) and stable across compaction.FileStore.Insert). A DB-enforced unique index removes the O(n) scan and
the race window.EnsureIndex(field, unique=true); a write violating
uniqueness returns ErrDuplicateKey. Builds on the existing secondary-index
machinery (sidx_*.json).engine/secondary_index.go, engine/collection.go, docs.sessions record with an existing name fails with a
typed duplicate error.UpdateStatusIf /
FinalizeExit (CAS on expected status) and ctxstore.CompareAndSet. The engine
has optimistic transactions but no ergonomic single-record CAS.rev, already implicit in the
latest-entry-wins model) and a conditional update:
UpdateIf(key, expectedRev, data) and/or predicate form
UpdateIf(key, func(cur) bool, data). Return (applied bool, err) so a
no-match is a clean no-op, matching warden's CAS semantics. Can be built over the
existing tx path or as a direct locked read-check-write.engine/collection.go, store/ndjson.go (expose rev/ts), docs.(false, nil).UpdateStatusIf and context CAS map 1:1 onto this
primitive with no application-side locking.closed/<id>.json).Upsert(key, data) — insert if absent, replace if present, one
locked op.engine/collection.go.Mostly satisfied by the release roadmap; called out here so the warden port has a tracked dependency.
List returns sessions sorted by UpdatedAt descending. At
warden's scale (dozens–hundreds active) an in-memory sort after an eq/prefix
scan is fine, but a directional order_by (release-roadmap Q2) makes it
first-class. No new item — depends on Q2.events, messages, spend, savings are queried by time window and by
owner id. Owner id is served by the existing eq secondary index; time-window
(ts >= X) wants the range index from release-roadmap Q3. No new item —
depends on Q3.Count(filter) and Exists(key) that use indexes when available
and short-circuit.engine/collection.go.SyncMode
none or interval (recommend interval, ~1s) so warden gets crash-torn-write
safety without per-write fsync cost; let warden opt into always per collection
(e.g. spend) if desired.scriva/engine defaults, docs/embedding.md.Watch events in-process — warden can subscribe
directly instead of re-scanning.Collection.Watch is usable purely in-process (no gRPC),
document the embedded subscription API, and honor the overflow signal
(release-roadmap D4, already shipped) so a slow warden consumer resyncs.docs/embedding.md, example.sessions/*.json, closed/*.json,
mailbox/*.json, context.json, ledgers on disk. Adoption must not lose them.Events into the events collection). ScrivaDB's contribution: a documented
bulk-load path (InsertMany, already present) and guidance in
docs/embedding.md. Optionally a small LoadJSONL(collection, reader) helper.docs/embedding.md; optional engine bulk-load helper.events/messages grow steadily. The existing
background compactor + rebalancer handle this, and on-demand compaction (F3)
lets warden force a pass (e.g. before snapshot). Verify compaction cadence
defaults are sane for many small collections. Mostly existing — depends on F3.Events slice and mailbox inboxes today.events/messages).RWMutex → matches warden's single-writer daemon; no extra
locking needed on warden's side.go get github.com/srjn45/scriva/engine). Not brew/apt/GHCR — those remain
for the standalone server/CLI.engine package (EMB-1, EMB-4) whose
import graph is minimal (EMB-3).Phase 1 (unblock): EMB-1 (public engine) ← nothing else can start without it
EMB-3 (dep hygiene) ← do alongside; cheap to keep clean
Phase 2 (fit): KEY-1 (string keys) ┐ the two real impedance mismatches
KEY-3 (CAS) ┘
EMB-2 (embedded façade)
Phase 3 (polish): KEY-2 (unique idx), KEY-4 (upsert), QRY-3 (count/exists),
OPS-1 (durability default), OPS-2 (in-proc Watch),
EMB-4 (semver + docs)
Phase 4 (migrate): OPS-3 (import) → then warden swaps FileStore → ScrivaStore
behind the existing store.Store interface
Depends on release roadmap: QRY-1→Q2, QRY-2→Q3, OPS-4→F2, OPS-5→F3
The warden side is a drop-in: internal/store.Store is already a clean
interface, so a ScrivaStore implementation lands behind it with no churn in
callers — the port is gated only on Phase 1–2 here.
EMB — Embeddable engine
internal/engine → public engine package 🔴Opendocs/embedding.mdKEY — Keys, uniqueness & CAS
QRY — Query fit
OPS — Embedded operations
Can you improve this documentation? These fine people already did:
Srajan Pathak & srjn45Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |