A SQLite target for the engine's snapshot seam
(vaelii.impl.io.snapshot) — a SnapshotSink that writes a KB image to a
single-file database and a SnapshotSource that reads it back.
This is the good SQLite lane. A live records/index store over a database
pays a round trip per probe, two orders of magnitude off local, so the query
path stays RAM-resident on every backend. A snapshot is the opposite
shape: an image is O(sections) bulk blob transfers, not O(records) tiny
probes. SQLite is where that shape is cheapest of all — the whole image is one
file with no server to stand up, so "put my KB in a file" — for backup, for
shipping a corpus to another host, for embedding a KB beside an app — is
answered here, and the file is the artifact you copy.
The seam is two ops each side (write-section!/commit!,
read-manifest/read-section) and the image is a set of named sections
plus a manifest. Here:
(image, section, seq, chunk blob) — each chunk a
nippy-frozen batch of frames, so a section of a million entries is a hundred
bulk inserts, not a million;read-manifest returns nil,
and the caller rebuilds. That is the file sink's "write manifest.edn
last" rule, but enforced by the database rather than by ordering: a failed
write rolls back the sections too, it does not leave half of them;kv/index-layout-version and the records fingerprint)
rides both a column (queryable) and the manifest blob, and the
validate-or-discard check is the shared snapshot/decision — a mismatched
image is discarded and the source rebuilds, never trusted.A section written through this sink reads back frame-identical through any source (file, memory, Postgres, SQLite) — the portability the seam exists to give.
Point the sink at a file-backed db-spec ({:dbtype "sqlite" :dbname "kb.db"} or a jdbc:sqlite:… url). The sink, the source, and
ensure-schema! each open their own connection, and for a file they all see
the same database — the same multi-connection shape the Postgres sink has. An
in-memory :memory: spec gives each connection its own database, so the
schema one opens is invisible to the next; use a file, or a shared-cache
datasource, if you need memory.
Apache-2.0, and an adapter: it depends on the SSPL engine's seam and is
never depended on by it. It implements vaelii.impl.io.snapshot's protocols,
the same way a record-store adapter implements vaelii.impl.protocols.
A SQLite target for the engine's snapshot seam
(`vaelii.impl.io.snapshot`) — a `SnapshotSink` that writes a KB image to a
single-file database and a `SnapshotSource` that reads it back.
This is the **good SQLite lane**. A live records/index store over a database
pays a round trip per probe, two orders of magnitude off local, so the query
path stays RAM-resident on every backend. A **snapshot** is the opposite
shape: an image is `O(sections)` bulk blob transfers, not `O(records)` tiny
probes. SQLite is where that shape is cheapest of all — the whole image is one
file with no server to stand up, so "put my KB in a file" — for backup, for
shipping a corpus to another host, for embedding a KB beside an app — is
answered here, and the file is the artifact you copy.
## The shape
The seam is two ops each side (`write-section!`/`commit!`,
`read-manifest`/`read-section`) and the image is a set of **named sections**
plus a **manifest**. Here:
* a section is a row stream `(image, section, seq, chunk blob)` — each chunk a
nippy-frozen batch of frames, so a section of a million entries is a hundred
bulk inserts, not a million;
* the whole image writes inside **one transaction** and the **manifest row
commits it** — so a crash leaves no manifest, `read-manifest` returns nil,
and the caller rebuilds. That is the file sink's "write `manifest.edn`
last" rule, but enforced by the database rather than by ordering: a failed
write rolls back the sections too, it does not leave half of them;
* the validity stamp (`kv/index-layout-version` and the records fingerprint)
rides both a **column** (queryable) and the manifest blob, and the
validate-or-discard check is the shared `snapshot/decision` — a mismatched
image is discarded and the source rebuilds, never trusted.
A section written through this sink reads back frame-identical through any
source (file, memory, Postgres, SQLite) — the portability the seam exists to
give.
## The file
Point the sink at a **file-backed** db-spec (`{:dbtype "sqlite" :dbname
"kb.db"}` or a `jdbc:sqlite:…` url). The sink, the source, and
`ensure-schema!` each open their own connection, and for a file they all see
the same database — the same multi-connection shape the Postgres sink has. An
in-memory `:memory:` spec gives **each connection its own database**, so the
schema one opens is invisible to the next; use a file, or a shared-cache
datasource, if you need memory.
## Boundary
Apache-2.0, and an **adapter**: it depends on the SSPL engine's seam and is
never depended on by it. It implements `vaelii.impl.io.snapshot`'s protocols,
the same way a record-store adapter implements `vaelii.impl.protocols`.(drop-image! ds image)Remove an image's sections and manifest. A no-op if the tables are absent.
Remove an image's sections and manifest. A no-op if the tables are absent.
(ensure-schema! ds)Create the two image tables if they are absent. Idempotent; runs in autocommit before a sink opens its transaction.
Create the two image tables if they are absent. Idempotent; runs in autocommit before a sink opens its transaction.
(sqlite-sink ds image)(sqlite-sink ds image {:keys [chunk-size] :or {chunk-size 10000}})A SnapshotSink writing an image to SQLite over ds (a next.jdbc db-spec or
datasource). Use a file-backed ds — the sink, the source and
ensure-schema! open separate connections, and only a file shares one database
across them (see the ns docstring on :memory:).
Opens one transaction; opening for an image clears any prior one, the same
fresh-image rule the file sink gets by deleting its manifest. commit!
commits the transaction; anything else must close the sink — it is
java.io.Closeable, and a close without a commit rolls the whole image back.
Use with-open.
:chunk-size frames per row (default 10000).
A `SnapshotSink` writing an `image` to SQLite over `ds` (a next.jdbc db-spec or datasource). Use a **file-backed** `ds` — the sink, the source and `ensure-schema!` open separate connections, and only a file shares one database across them (see the ns docstring on `:memory:`). Opens one transaction; opening for an image clears any prior one, the same fresh-image rule the file sink gets by deleting its manifest. **`commit!` commits the transaction; anything else must `close` the sink** — it is `java.io.Closeable`, and a close without a commit rolls the whole image back. Use `with-open`. `:chunk-size` frames per row (default 10000).
(sqlite-source ds image)A read-only SnapshotSource over the image a sqlite-sink committed on ds.
Reads nothing until asked — read-manifest for the validity check,
read-section for the install.
A read-only `SnapshotSource` over the `image` a `sqlite-sink` committed on `ds`. Reads nothing until asked — `read-manifest` for the validity check, `read-section` for the install.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |