A Postgres target for the engine's record store seam
(vaelii.impl.protocols/RecordStore) — the durable ground truth a KB is recovered
from: canonical sentexes and justifications, keyed by integer handle, in a database
an operator already runs.
It buys COPY (copy-sentexes! below, the fastest ingest path any records backend
has), an operator's existing backup, PITR, replication, monitoring and access
control, a store larger than one machine's disk, and a real REPEATABLE READ
snapshot while something else is writing hard.
It does not buy a shared KB. Belief lives in the writing process's RAM — the
JTMS and the taxonomy closures — so a second process connected to this same database
does not see the first's beliefs, and its retraction sweep deletes records the first
still believes. That is the engine's single-writer contract, and a server does not
weaken it by one clause (docs/storage.md, The single-writer contract).
Three tables, and the handle is the key throughout:
vaelii_record (id, kind, frame, premise, strength) — one row per record, kind
splitting sentexes (0) from justifications (1), frame the whole record
nippy-frozen so a fetch thaws back type-identical (an AtomicSentex stays an
AtomicSentex). id is bigint: handles are ints in the engine today, and a
column type is the one place that decision becomes a migration on a table with
100M rows in it.strength column as the authoritative
value and is assoced back onto the thawed record on read — so mark-premise is a
one-row column update rather than a frame rewrite, and the column and the frame
cannot drift because the column always wins.vaelii_record_provenance (id, prov) and vaelii_record_meta (k, v), the latter
holding high_water — the largest handle ever allocated, so a delete of the max
handle followed by a reopen still never reissues it.A put is one statement, not a transaction: a data-modifying CTE writes the record
and moves high_water together, so a write costs one round trip rather than a
BEGIN/two statements/COMMIT pair of four. A delete pairs the record and its
provenance the same way.
A point read is a network round trip where the disk store's is a page touch, and everything here is a response to that:
premise-ids walk itself — the walk
already selects the column, and recover asks for every one of those strengths
immediately afterwards, so the pair costs one scan instead of one scan plus a round
trip per premise;reduce-rows): autocommit
off and a set fetch size, which is what makes the driver hand back a portal instead
of buffering the whole result.Point the store at a db-spec ({:dbtype "postgresql" :host … :dbname …}, or
{:jdbcUrl "jdbc:postgresql://…"}) and it builds and owns a HikariCP pool — one
writer and N readers, by the contract above. Hand it a javax.sql.DataSource
instead and it borrows from that and closes nothing. :schema puts the three tables
in a schema of their own, so one database can hold several KBs and an operator can
drop one with DROP SCHEMA.
fsync is a no-op with a reason: durability here is the server's fsync, settled by
its synchronous_commit and its WAL, and there is no client-side buffer for this
store to force. It registers its close with the engine's durability daemon all
the same, so a JVM that exits without a close! still releases the pool.
Apache-2.0, and an adapter: it implements the SSPL engine's
vaelii.impl.protocols/RecordStore and is never depended on by it. Core wires it in
by a lazy requiring-resolve, so the engine never loads a JDBC driver unless a KB
selects :pg-memory or :pg-disk.
A Postgres target for the engine's **record store** seam
(`vaelii.impl.protocols/RecordStore`) — the durable ground truth a KB is recovered
from: canonical sentexes and justifications, keyed by integer handle, in a database
an operator already runs.
## What a server buys, and what it does not
It buys `COPY` (`copy-sentexes!` below, the fastest ingest path any records backend
has), an operator's existing backup, PITR, replication, monitoring and access
control, a store larger than one machine's disk, and a real `REPEATABLE READ`
snapshot while something else is writing hard.
It does **not** buy a shared KB. Belief lives in the writing process's RAM — the
JTMS and the taxonomy closures — so a second process connected to this same database
does not see the first's beliefs, and its retraction sweep deletes records the first
still believes. That is the engine's single-writer contract, and a server does not
weaken it by one clause (`docs/storage.md`, *The single-writer contract*).
## The shape
Three tables, and the handle is the key throughout:
* `vaelii_record (id, kind, frame, premise, strength)` — one row per record, `kind`
splitting sentexes (0) from justifications (1), `frame` the **whole record**
nippy-frozen so a fetch thaws back type-identical (an `AtomicSentex` stays an
`AtomicSentex`). `id` is **`bigint`**: handles are ints in the engine today, and a
column type is the one place that decision becomes a migration on a table with
100M rows in it.
* a sentex's **assumption strength** rides the `strength` column as the authoritative
value and is `assoc`ed back onto the thawed record on read — so `mark-premise` is a
one-row column update rather than a frame rewrite, and the column and the frame
cannot drift because the column always wins.
* `vaelii_record_provenance (id, prov)` and `vaelii_record_meta (k, v)`, the latter
holding `high_water` — the largest handle ever allocated, so a delete of the max
handle followed by a reopen still never reissues it.
A put is **one statement**, not a transaction: a data-modifying CTE writes the record
and moves `high_water` together, so a write costs one round trip rather than a
`BEGIN`/two statements/`COMMIT` pair of four. A delete pairs the record and its
provenance the same way.
## The round trip, and what is done about it
A point read is a network round trip where the disk store's is a page touch, and
everything here is a response to that:
* a **fetch LRU per kind** in front of every read, so a hot handle never reaches the
server;
* a **premise-strength cache** filled by the `premise-ids` walk itself — the walk
already selects the column, and `recover` asks for every one of those strengths
immediately afterwards, so the pair costs one scan instead of one scan plus a round
trip per premise;
* **enumeration streams** through a server-side cursor (`reduce-rows`): autocommit
off and a set fetch size, which is what makes the driver hand back a portal instead
of buffering the whole result.
## Connections
Point the store at a db-spec (`{:dbtype "postgresql" :host … :dbname …}`, or
`{:jdbcUrl "jdbc:postgresql://…"}`) and it builds and owns a HikariCP pool — one
writer and N readers, by the contract above. Hand it a `javax.sql.DataSource`
instead and it borrows from that and closes nothing. `:schema` puts the three tables
in a schema of their own, so one database can hold several KBs and an operator can
drop one with `DROP SCHEMA`.
`fsync` is a no-op with a reason: durability here is the server's `fsync`, settled by
its `synchronous_commit` and its WAL, and there is no client-side buffer for this
store to force. It registers its **close** with the engine's durability daemon all
the same, so a JVM that exits without a `close!` still releases the pool.
## Boundary
Apache-2.0, and an **adapter**: it implements the SSPL engine's
`vaelii.impl.protocols/RecordStore` and is never depended on by it. Core wires it in
by a lazy `requiring-resolve`, so the engine never loads a JDBC driver unless a KB
selects `:pg-memory` or `:pg-disk`.A Postgres target for the engine's snapshot seam
(vaelii.impl.io.snapshot) — a SnapshotSink that writes a KB image to a
database and a SnapshotSource that reads it back.
This is the good Postgres lane. A live records/index store over Postgres
pays a round trip per probe, two orders of magnitude off local, so the query
path stays RAM-resident on every backend. A snapshot is the opposite
shape: an image is O(sections) bulk blob transfers, not O(records) tiny
probes, and bulk transfer over a connection is what a server does well. "Put
my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to
another host — is answered here, where the server carries the image and an
operator's existing backup and replication carry it for free.
The seam is two ops each side (write-section!/commit!,
read-manifest/read-section) and the image is a set of named sections
plus a manifest. Here:
(image, section, seq, chunk bytea) — each chunk a
nippy-frozen batch of frames, so a section of a million entries is a hundred
bulk inserts, not a million;read-manifest returns nil,
and the caller rebuilds. That is the file sink's "write manifest.edn
last" rule, but enforced by the database rather than by ordering: a failed
write rolls back the sections too, it does not leave half of them;kv/index-layout-version and the records fingerprint)
rides both a column (queryable) and the manifest blob, and the
validate-or-discard check is the shared snapshot/decision — a mismatched
image is discarded and the source rebuilds, never trusted.A section written through this sink reads back frame-identical through any source (file, memory, Postgres) — the portability the seam exists to give.
Apache-2.0, and an adapter: it depends on the SSPL engine's seam and is
never depended on by it. It implements vaelii.impl.io.snapshot's protocols,
the same way a record-store adapter implements vaelii.impl.protocols.
A Postgres target for the engine's snapshot seam (`vaelii.impl.io.snapshot`) — a `SnapshotSink` that writes a KB image to a database and a `SnapshotSource` that reads it back. This is the **good Postgres lane**. A live records/index store over Postgres pays a round trip per probe, two orders of magnitude off local, so the query path stays RAM-resident on every backend. A **snapshot** is the opposite shape: an image is `O(sections)` bulk blob transfers, not `O(records)` tiny probes, and bulk transfer over a connection is what a server does well. "Put my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to another host — is answered here, where the server carries the image and an operator's existing backup and replication carry it for free. ## The shape The seam is two ops each side (`write-section!`/`commit!`, `read-manifest`/`read-section`) and the image is a set of **named sections** plus a **manifest**. Here: * a section is a row stream `(image, section, seq, chunk bytea)` — each chunk a nippy-frozen batch of frames, so a section of a million entries is a hundred bulk inserts, not a million; * the whole image writes inside **one transaction** and the **manifest row commits it** — so a crash leaves no manifest, `read-manifest` returns nil, and the caller rebuilds. That is the file sink's "write `manifest.edn` last" rule, but enforced by the database rather than by ordering: a failed write rolls back the sections too, it does not leave half of them; * the validity stamp (`kv/index-layout-version` and the records fingerprint) rides both a **column** (queryable) and the manifest blob, and the validate-or-discard check is the shared `snapshot/decision` — a mismatched image is discarded and the source rebuilds, never trusted. A section written through this sink reads back frame-identical through any source (file, memory, Postgres) — the portability the seam exists to give. ## Boundary Apache-2.0, and an **adapter**: it depends on the SSPL engine's seam and is never depended on by it. It implements `vaelii.impl.io.snapshot`'s protocols, the same way a record-store adapter implements `vaelii.impl.protocols`.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |