A Postgres target for the engine's snapshot seam
(vaelii.impl.io.snapshot) — a SnapshotSink that writes a KB image to a
database and a SnapshotSource that reads it back.
This is the good Postgres lane. A live records/index store over Postgres
pays a round trip per probe, two orders of magnitude off local, so the query
path stays RAM-resident on every backend. A snapshot is the opposite
shape: an image is O(sections) bulk blob transfers, not O(records) tiny
probes, and bulk transfer over a connection is what a server does well. "Put
my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to
another host — is answered here, where the server carries the image and an
operator's existing backup and replication carry it for free.
The seam is two ops each side (write-section!/commit!,
read-manifest/read-section) and the image is a set of named sections
plus a manifest. Here:
(image, section, seq, chunk bytea) — each chunk a
nippy-frozen batch of frames, so a section of a million entries is a hundred
bulk inserts, not a million;read-manifest returns nil,
and the caller rebuilds. That is the file sink's "write manifest.edn
last" rule, but enforced by the database rather than by ordering: a failed
write rolls back the sections too, it does not leave half of them;kv/index-layout-version and the records fingerprint)
rides both a column (queryable) and the manifest blob, and the
validate-or-discard check is the shared snapshot/decision — a mismatched
image is discarded and the source rebuilds, never trusted.A section written through this sink reads back frame-identical through any source (file, memory, Postgres) — the portability the seam exists to give.
Apache-2.0, and an adapter: it depends on the SSPL engine's seam and is
never depended on by it. It implements vaelii.impl.io.snapshot's protocols,
the same way a record-store adapter implements vaelii.impl.protocols.
A Postgres target for the engine's snapshot seam (`vaelii.impl.io.snapshot`) — a `SnapshotSink` that writes a KB image to a database and a `SnapshotSource` that reads it back. This is the **good Postgres lane**. A live records/index store over Postgres pays a round trip per probe, two orders of magnitude off local, so the query path stays RAM-resident on every backend. A **snapshot** is the opposite shape: an image is `O(sections)` bulk blob transfers, not `O(records)` tiny probes, and bulk transfer over a connection is what a server does well. "Put my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to another host — is answered here, where the server carries the image and an operator's existing backup and replication carry it for free. ## The shape The seam is two ops each side (`write-section!`/`commit!`, `read-manifest`/`read-section`) and the image is a set of **named sections** plus a **manifest**. Here: * a section is a row stream `(image, section, seq, chunk bytea)` — each chunk a nippy-frozen batch of frames, so a section of a million entries is a hundred bulk inserts, not a million; * the whole image writes inside **one transaction** and the **manifest row commits it** — so a crash leaves no manifest, `read-manifest` returns nil, and the caller rebuilds. That is the file sink's "write `manifest.edn` last" rule, but enforced by the database rather than by ordering: a failed write rolls back the sections too, it does not leave half of them; * the validity stamp (`kv/index-layout-version` and the records fingerprint) rides both a **column** (queryable) and the manifest blob, and the validate-or-discard check is the shared `snapshot/decision` — a mismatched image is discarded and the source rebuilds, never trusted. A section written through this sink reads back frame-identical through any source (file, memory, Postgres) — the portability the seam exists to give. ## Boundary Apache-2.0, and an **adapter**: it depends on the SSPL engine's seam and is never depended on by it. It implements `vaelii.impl.io.snapshot`'s protocols, the same way a record-store adapter implements `vaelii.impl.protocols`.
(drop-image! ds image)Remove an image's sections and manifest. A no-op if the tables are absent.
Remove an image's sections and manifest. A no-op if the tables are absent.
(ensure-schema! ds)Create the two image tables if they are absent. Idempotent; runs in autocommit before a sink opens its transaction.
Create the two image tables if they are absent. Idempotent; runs in autocommit before a sink opens its transaction.
(pg-sink ds image)(pg-sink ds image {:keys [chunk-size] :or {chunk-size 10000}})A SnapshotSink writing an image to Postgres over ds (a next.jdbc db-spec
or datasource). Opens one transaction; opening for an image clears any prior
one, the same fresh-image rule the file sink gets by deleting its manifest.
commit! commits the transaction; anything else must close the sink —
it is java.io.Closeable, and a close without a commit rolls the whole image
back. Use with-open.
:chunk-size frames per row (default 10000).
A `SnapshotSink` writing an `image` to Postgres over `ds` (a next.jdbc db-spec or datasource). Opens one transaction; opening for an image clears any prior one, the same fresh-image rule the file sink gets by deleting its manifest. **`commit!` commits the transaction; anything else must `close` the sink** — it is `java.io.Closeable`, and a close without a commit rolls the whole image back. Use `with-open`. `:chunk-size` frames per row (default 10000).
(pg-source ds image)A read-only SnapshotSource over the image a pg-sink committed on ds.
Reads nothing until asked — read-manifest for the validity check,
read-section for the install.
A read-only `SnapshotSource` over the `image` a `pg-sink` committed on `ds`. Reads nothing until asked — `read-manifest` for the validity check, `read-section` for the install.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |