Liking cljdoc? Tell your friends :D

vaelii.postgres.record-store

A Postgres target for the engine's record store seam (vaelii.impl.protocols/RecordStore) — the durable ground truth a KB is recovered from: canonical sentexes and justifications, keyed by integer handle, in a database an operator already runs.

What a server buys, and what it does not

It buys COPY (copy-sentexes! below, the fastest ingest path any records backend has), an operator's existing backup, PITR, replication, monitoring and access control, a store larger than one machine's disk, and a real REPEATABLE READ snapshot while something else is writing hard.

It does not buy a shared KB. Belief lives in the writing process's RAM — the JTMS and the taxonomy closures — so a second process connected to this same database does not see the first's beliefs, and its retraction sweep deletes records the first still believes. That is the engine's single-writer contract, and a server does not weaken it by one clause (docs/storage.md, The single-writer contract).

The shape

Three tables, and the handle is the key throughout:

  • vaelii_record (id, kind, frame, premise, strength) — one row per record, kind splitting sentexes (0) from justifications (1), frame the whole record nippy-frozen so a fetch thaws back type-identical (an AtomicSentex stays an AtomicSentex). id is bigint: handles are ints in the engine today, and a column type is the one place that decision becomes a migration on a table with 100M rows in it.
  • a sentex's assumption strength rides the strength column as the authoritative value and is assoced back onto the thawed record on read — so mark-premise is a one-row column update rather than a frame rewrite, and the column and the frame cannot drift because the column always wins.
  • vaelii_record_provenance (id, prov) and vaelii_record_meta (k, v), the latter holding high_water — the largest handle ever allocated, so a delete of the max handle followed by a reopen still never reissues it.

A put is one statement, not a transaction: a data-modifying CTE writes the record and moves high_water together, so a write costs one round trip rather than a BEGIN/two statements/COMMIT pair of four. A delete pairs the record and its provenance the same way.

The round trip, and what is done about it

A point read is a network round trip where the disk store's is a page touch, and everything here is a response to that:

  • a fetch LRU per kind in front of every read, so a hot handle never reaches the server;
  • a premise-strength cache filled by the premise-ids walk itself — the walk already selects the column, and recover asks for every one of those strengths immediately afterwards, so the pair costs one scan instead of one scan plus a round trip per premise;
  • enumeration streams through a server-side cursor (reduce-rows): autocommit off and a set fetch size, which is what makes the driver hand back a portal instead of buffering the whole result.

Connections

Point the store at a db-spec ({:dbtype "postgresql" :host … :dbname …}, or {:jdbcUrl "jdbc:postgresql://…"}) and it builds and owns a HikariCP pool — one writer and N readers, by the contract above. Hand it a javax.sql.DataSource instead and it borrows from that and closes nothing. :schema puts the three tables in a schema of their own, so one database can hold several KBs and an operator can drop one with DROP SCHEMA.

fsync is a no-op with a reason: durability here is the server's fsync, settled by its synchronous_commit and its WAL, and there is no client-side buffer for this store to force. It registers its close with the engine's durability daemon all the same, so a JVM that exits without a close! still releases the pool.

Boundary

Apache-2.0, and an adapter: it implements the SSPL engine's vaelii.impl.protocols/RecordStore and is never depended on by it. Core wires it in by a lazy requiring-resolve, so the engine never loads a JDBC driver unless a KB selects :pg-memory or :pg-disk.

A Postgres target for the engine's **record store** seam
(`vaelii.impl.protocols/RecordStore`) — the durable ground truth a KB is recovered
from: canonical sentexes and justifications, keyed by integer handle, in a database
an operator already runs.

## What a server buys, and what it does not

It buys `COPY` (`copy-sentexes!` below, the fastest ingest path any records backend
has), an operator's existing backup, PITR, replication, monitoring and access
control, a store larger than one machine's disk, and a real `REPEATABLE READ`
snapshot while something else is writing hard.

It does **not** buy a shared KB.  Belief lives in the writing process's RAM — the
JTMS and the taxonomy closures — so a second process connected to this same database
does not see the first's beliefs, and its retraction sweep deletes records the first
still believes.  That is the engine's single-writer contract, and a server does not
weaken it by one clause (`docs/storage.md`, *The single-writer contract*).

## The shape

Three tables, and the handle is the key throughout:

* `vaelii_record (id, kind, frame, premise, strength)` — one row per record, `kind`
  splitting sentexes (0) from justifications (1), `frame` the **whole record**
  nippy-frozen so a fetch thaws back type-identical (an `AtomicSentex` stays an
  `AtomicSentex`).  `id` is **`bigint`**: handles are ints in the engine today, and a
  column type is the one place that decision becomes a migration on a table with
  100M rows in it.
* a sentex's **assumption strength** rides the `strength` column as the authoritative
  value and is `assoc`ed back onto the thawed record on read — so `mark-premise` is a
  one-row column update rather than a frame rewrite, and the column and the frame
  cannot drift because the column always wins.
* `vaelii_record_provenance (id, prov)` and `vaelii_record_meta (k, v)`, the latter
  holding `high_water` — the largest handle ever allocated, so a delete of the max
  handle followed by a reopen still never reissues it.

A put is **one statement**, not a transaction: a data-modifying CTE writes the record
and moves `high_water` together, so a write costs one round trip rather than a
`BEGIN`/two statements/`COMMIT` pair of four.  A delete pairs the record and its
provenance the same way.

## The round trip, and what is done about it

A point read is a network round trip where the disk store's is a page touch, and
everything here is a response to that:

* a **fetch LRU per kind** in front of every read, so a hot handle never reaches the
  server;
* a **premise-strength cache** filled by the `premise-ids` walk itself — the walk
  already selects the column, and `recover` asks for every one of those strengths
  immediately afterwards, so the pair costs one scan instead of one scan plus a round
  trip per premise;
* **enumeration streams** through a server-side cursor (`reduce-rows`): autocommit
  off and a set fetch size, which is what makes the driver hand back a portal instead
  of buffering the whole result.

## Connections

Point the store at a db-spec (`{:dbtype "postgresql" :host … :dbname …}`, or
`{:jdbcUrl "jdbc:postgresql://…"}`) and it builds and owns a HikariCP pool — one
writer and N readers, by the contract above.  Hand it a `javax.sql.DataSource`
instead and it borrows from that and closes nothing.  `:schema` puts the three tables
in a schema of their own, so one database can hold several KBs and an operator can
drop one with `DROP SCHEMA`.

`fsync` is a no-op with a reason: durability here is the server's `fsync`, settled by
its `synchronous_commit` and its WAL, and there is no client-side buffer for this
store to force.  It registers its **close** with the engine's durability daemon all
the same, so a JVM that exits without a `close!` still releases the pool.

## Boundary

Apache-2.0, and an **adapter**: it implements the SSPL engine's
`vaelii.impl.protocols/RecordStore` and is never depended on by it.  Core wires it in
by a lazy `requiring-resolve`, so the engine never loads a JDBC driver unless a KB
selects `:pg-memory` or `:pg-disk`.
raw docstring

vaelii.postgres.snapshot

A Postgres target for the engine's snapshot seam (vaelii.impl.io.snapshot) — a SnapshotSink that writes a KB image to a database and a SnapshotSource that reads it back.

This is the good Postgres lane. A live records/index store over Postgres pays a round trip per probe, two orders of magnitude off local, so the query path stays RAM-resident on every backend. A snapshot is the opposite shape: an image is O(sections) bulk blob transfers, not O(records) tiny probes, and bulk transfer over a connection is what a server does well. "Put my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to another host — is answered here, where the server carries the image and an operator's existing backup and replication carry it for free.

The shape

The seam is two ops each side (write-section!/commit!, read-manifest/read-section) and the image is a set of named sections plus a manifest. Here:

  • a section is a row stream (image, section, seq, chunk bytea) — each chunk a nippy-frozen batch of frames, so a section of a million entries is a hundred bulk inserts, not a million;
  • the whole image writes inside one transaction and the manifest row commits it — so a crash leaves no manifest, read-manifest returns nil, and the caller rebuilds. That is the file sink's "write manifest.edn last" rule, but enforced by the database rather than by ordering: a failed write rolls back the sections too, it does not leave half of them;
  • the validity stamp (kv/index-layout-version and the records fingerprint) rides both a column (queryable) and the manifest blob, and the validate-or-discard check is the shared snapshot/decision — a mismatched image is discarded and the source rebuilds, never trusted.

A section written through this sink reads back frame-identical through any source (file, memory, Postgres) — the portability the seam exists to give.

Boundary

Apache-2.0, and an adapter: it depends on the SSPL engine's seam and is never depended on by it. It implements vaelii.impl.io.snapshot's protocols, the same way a record-store adapter implements vaelii.impl.protocols.

A Postgres target for the engine's snapshot seam
(`vaelii.impl.io.snapshot`) — a `SnapshotSink` that writes a KB image to a
database and a `SnapshotSource` that reads it back.

This is the **good Postgres lane**.  A live records/index store over Postgres
pays a round trip per probe, two orders of magnitude off local, so the query
path stays RAM-resident on every backend.  A **snapshot** is the opposite
shape: an image is `O(sections)` bulk blob transfers, not `O(records)` tiny
probes, and bulk transfer over a connection is what a server does well.  "Put
my KB in Postgres" — for backup, replication, PITR, or shipping a corpus to
another host — is answered here, where the server carries the image and an
operator's existing backup and replication carry it for free.

## The shape

The seam is two ops each side (`write-section!`/`commit!`,
`read-manifest`/`read-section`) and the image is a set of **named sections**
plus a **manifest**.  Here:

* a section is a row stream `(image, section, seq, chunk bytea)` — each chunk a
  nippy-frozen batch of frames, so a section of a million entries is a hundred
  bulk inserts, not a million;
* the whole image writes inside **one transaction** and the **manifest row
  commits it** — so a crash leaves no manifest, `read-manifest` returns nil,
  and the caller rebuilds.  That is the file sink's "write `manifest.edn`
  last" rule, but enforced by the database rather than by ordering: a failed
  write rolls back the sections too, it does not leave half of them;
* the validity stamp (`kv/index-layout-version` and the records fingerprint)
  rides both a **column** (queryable) and the manifest blob, and the
  validate-or-discard check is the shared `snapshot/decision` — a mismatched
  image is discarded and the source rebuilds, never trusted.

A section written through this sink reads back frame-identical through any
source (file, memory, Postgres) — the portability the seam exists to give.

## Boundary

Apache-2.0, and an **adapter**: it depends on the SSPL engine's seam and is
never depended on by it.  It implements `vaelii.impl.io.snapshot`'s protocols,
the same way a record-store adapter implements `vaelii.impl.protocols`.
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
Move to previous article
Move to next article
Ctrl+/Jump to the search field
× close