vaelii.postgres.record-store — com.vaelii/postgres 0.12.0

A Postgres target for the engine's record store seam (vaelii.impl.protocols/RecordStore) — the durable ground truth a KB is recovered from: canonical sentexes and justifications, keyed by integer handle, in a database an operator already runs.

What a server buys, and what it does not

It buys COPY (copy-sentexes! below, the fastest ingest path any records backend has), an operator's existing backup, PITR, replication, monitoring and access control, a store larger than one machine's disk, and a real REPEATABLE READ snapshot while something else is writing hard.

It does not buy a shared KB. Belief lives in the writing process's RAM — the JTMS and the taxonomy closures — so a second process connected to this same database does not see the first's beliefs, and its retraction sweep deletes records the first still believes. That is the engine's single-writer contract, and a server does not weaken it by one clause (docs/storage.md, The single-writer contract).

The shape

Three tables, and the handle is the key throughout:

• vaelii_record (id, kind, frame, premise, strength) — one row per record, kind splitting sentexes (0) from justifications (1), frame the whole record nippy-frozen so a fetch thaws back type-identical (an AtomicSentex stays an AtomicSentex). id is bigint: handles are ints in the engine today, and a column type is the one place that decision becomes a migration on a table with 100M rows in it.
• a sentex's assumption strength rides the strength column as the authoritative value and is assoced back onto the thawed record on read — so mark-premise is a one-row column update rather than a frame rewrite, and the column and the frame cannot drift because the column always wins.
• vaelii_record_provenance (id, prov) and vaelii_record_meta (k, v), the latter holding high_water — the largest handle ever allocated, so a delete of the max handle followed by a reopen still never reissues it.

A put is one statement, not a transaction: a data-modifying CTE writes the record and moves high_water together, so a write costs one round trip rather than a BEGIN/two statements/COMMIT pair of four. A delete pairs the record and its provenance the same way.

The round trip, and what is done about it

A point read is a network round trip where the disk store's is a page touch, and everything here is a response to that:

• a fetch LRU per kind in front of every read, so a hot handle never reaches the server;
• a premise-strength cache filled by the premise-ids walk itself — the walk already selects the column, and recover asks for every one of those strengths immediately afterwards, so the pair costs one scan instead of one scan plus a round trip per premise;
• enumeration streams through a server-side cursor (reduce-rows): autocommit off and a set fetch size, which is what makes the driver hand back a portal instead of buffering the whole result.

Connections

Point the store at a db-spec ({:dbtype "postgresql" :host … :dbname …}, or {:jdbcUrl "jdbc:postgresql://…"}) and it builds and owns a HikariCP pool — one writer and N readers, by the contract above. Hand it a javax.sql.DataSource instead and it borrows from that and closes nothing. :schema puts the three tables in a schema of their own, so one database can hold several KBs and an operator can drop one with DROP SCHEMA.

fsync is a no-op with a reason: durability here is the server's fsync, settled by its synchronous_commit and its WAL, and there is no client-side buffer for this store to force. It registers its close with the engine's durability daemon all the same, so a JVM that exits without a close! still releases the pool.

Boundary

Apache-2.0, and an adapter: it implements the SSPL engine's vaelii.impl.protocols/RecordStore and is never depended on by it. Core wires it in by a lazy requiring-resolve, so the engine never loads a JDBC driver unless a KB selects :pg-memory or :pg-disk.

copy-justifications! clj

(copy-justifications! store justifications)

(copy-justifications! store
                      justifications
                      {:keys [batch] :or {batch default-copy-batch}})

Bulk-load justifications — each carrying its :id — the same way copy-sentexes! loads sentexes. A justification has no strength column.

copy-sentexes! clj

(copy-sentexes! store sentexes)

(copy-sentexes! store
                sentexes
                {:keys [batch premises?]
                 :or {batch default-copy-batch premises? true}})

Bulk-load sentexes — each carrying its :id — through COPY … FROM STDIN BINARY, returning how many rows landed. This is the ingest path a server has and the others do not, and it is a load rather than an upsert: a handle the store already holds raises rather than being overwritten.

A record's :strength rosters it as a premise here, where put-sentex leaves the mark to mark-premise. The two doors are asked different questions: the engine stores a fact and marks it at the choke point one line later, while a bulk load has no such line — the strength a dump's record carries is what a later recover reads as its assumption, and re-asking for it a handle at a time is a round trip per record. {:premises? false} loads the strengths without the marks, for records that are derivations rather than assumptions. :batch rows per buffer (default 10000).

This is the seam's own front door, for an application holding a corpus already. The engine reaches the same path without knowing this namespace exists, through protocols/BulkLoading — which is what import! writes its records through.

ensure-schema! clj

(ensure-schema! ds sql)

Create the record, provenance and meta tables (and the premise index), and the schema itself when one is named. Idempotent; runs before any op.

pg-record-store clj

(pg-record-store spec)

(pg-record-store spec opts)

A durable RecordStore over Postgres.

spec is a next.jdbc db-spec ({:dbtype "postgresql" :host … :dbname … :user …} or {:jdbcUrl "jdbc:postgresql://…"}), in which case this builds and owns a HikariCP pool — or an existing javax.sql.DataSource, which it borrows and does not close. Its own keys, which never reach the driver, are store-keys:

• :schema — put the three tables in a schema of their own (created if absent), so one database holds several KBs and DROP SCHEMA removes one whole;
• :cache-capacity — the per-kind fetch LRU (default 65536);
• :premise-cache-capacity — how many premise strengths to hold in RAM (default 4,000,000); past it the walk stops filling and premise-strength pays a round trip;
• :fetch-size — rows per cursor fetch on the enumerations (default 5000);
• :pool-size — maximum pooled connections (default 8);
• :prefetch — act on a caller's prefetch hint (default true). Whether a hint ever arrives is the caller's setting (resolution/*prefetch-candidates*, off by default); this says whether this store does anything when one does, and false here is the hard off for benchmarking one against the other;
• :prefetch-min — how many handles in a hint must be uncached before a batched read is issued instead of leaving them to the point reads (default 4).

Creates the schema if absent and loads the handle counter, so a reopen mints above every handle the store has ever held. The store is java.io.Closeable; close releases the pool it built and drops its durability registration.

reduce-rows clj

(reduce-rows store sql params rf init)

Reduce rf over the rows sql selects, streaming them through a server-side cursor rather than buffering the result: the walk borrows a connection, turns autocommit off and sets a fetch size, which together are what make the driver hand back a portal. Without both, SELECT id FROM vaelii_record on a large store reads every row into the driver before the first one is visible.

rf is an ordinary reducing fn (rf acc ^ResultSet rs), called with the cursor positioned on each row; wrapping the accumulator in reduced stops the walk and closes the cursor without draining it. The transaction is rolled back on the way out — every walk here is a read — and the connection goes back to the pool.

store-keys clj

The keys pg-record-store reads itself. Everything else in the spec is the database's — :dbtype, :host, :dbname, :jdbcUrl, credentials — and is handed to the driver untouched, so this is the set that must be stripped before it is.