The operation log: each public write a KB takes, recorded as the call that made it.
A frame holds the operation's name, its arguments, and the inputs the call reads that
its arguments do not carry — the clock :created is stamped from, the creator, and the
dynamic bindings that change what a write stores. vaelii.core names the operations
and their classes in its write-ops table, routes each through run-op, and installs
the table as the dispatch replay! runs frames through.
A write the engine makes inside another write is part of the enclosing call: the
asserts inside assert-many, the assert a skolem mint makes, the retraction
retract! makes of an orphaned NAT. *in-op?* is true for the extent of a recorded
operation, and run-op appends no frame while it is; the nested write runs under the
bindings the enclosing operation made.
:replay — the call is recorded, and its arguments and inputs determine what it
stores.:seal — import!, clear!, recover, reindex, load-text!: the call's effect
depends on something no frame carries (a directory's contents, the records as a
whole). It marks the log unusable and requests a seal, which runs when it returns.:config — set-solver, add-prover, add-evaluatable, add-reasoner: the call
registers code, which no frame can carry, so it marks the log unusable.A log is unusable from the first write it cannot reproduce until a seal starts it
again (rotate!). mark-unusable! appends an {:unusable reason} frame, so the mark
outlives the process. A :seal or :config operation marks it, and so do arguments
nippy cannot freeze, a write a change-feed listener makes (a listener runs part-way
through another operation's settle, where vaelii.core/dispatch-feed! binds
*listener?*), and a record write outside every operation (LoggedRecords).
attach wraps a KB's record store in LoggedRecords, with a watermark: one above
every handle the store held when the log's generation began. The page cache can write
a record to disk before it writes the frame describing the operation that stored it,
so a crash can leave a record no durable frame accounts for. A replay finds every such
record at a handle at or above its own allocations, except a write to a handle below
the watermark — a deletion, a premise mark, a provenance change of a record the
generation began with. So before the first such write in an operation,
LoggedRecords fsyncs the operation's frame. An operation that only adds records
pays no fsync.
In replay mode (attach-replaying) the store allocates handles from the watermark
and checks each write against the record already stored at its handle: an equal record
is left as it is, a missing one is written, and a different one throws with
::diverged in its ex-data.
The file's first frame is a header naming the log's generation. rotate! starts
a generation: it truncates the file to a new header and clears the unusable mark.
replay! runs each operation frame through the installed dispatch with *replaying?*
bound, so run-op appends nothing and binds the frame's inputs, and with the change
feed off, since listeners belong to the process that made the writes. An operation
that refused when it was made refuses again, and replay goes on to the next frame.
vaelii.impl.seal decides which generation a restore replays and what a seal writes.
One file, <dir>/oplog/ops.log, of length-prefixed nippy frames in
vaelii.impl.disk.files' format. The durability daemon fsyncs it on its tick and
closes it on shutdown. An open truncates a torn trailing frame, as the record logs'
opens do.
The **operation log**: each public write a KB takes, recorded as the call that made it.
A frame holds the operation's name, its arguments, and the inputs the call reads that
its arguments do not carry — the clock `:created` is stamped from, the creator, and the
dynamic bindings that change what a write stores. `vaelii.core` names the operations
and their classes in its `write-ops` table, routes each through `run-op`, and installs
the table as the dispatch `replay!` runs frames through.
## One frame per outermost write
A write the engine makes inside another write is part of the enclosing call: the
asserts inside `assert-many`, the assert a skolem mint makes, the retraction
`retract!` makes of an orphaned NAT. `*in-op?*` is true for the extent of a recorded
operation, and `run-op` appends no frame while it is; the nested write runs under the
bindings the enclosing operation made.
## Classes
- `:replay` — the call is recorded, and its arguments and inputs determine what it
stores.
- `:seal` — `import!`, `clear!`, `recover`, `reindex`, `load-text!`: the call's effect
depends on something no frame carries (a directory's contents, the records as a
whole). It marks the log unusable and requests a seal, which runs when it returns.
- `:config` — `set-solver`, `add-prover`, `add-evaluatable`, `add-reasoner`: the call
registers code, which no frame can carry, so it marks the log unusable.
## Unusable
A log is **unusable** from the first write it cannot reproduce until a seal starts it
again (`rotate!`). `mark-unusable!` appends an `{:unusable reason}` frame, so the mark
outlives the process. A `:seal` or `:config` operation marks it, and so do arguments
nippy cannot freeze, a write a change-feed listener makes (a listener runs part-way
through another operation's settle, where `vaelii.core/dispatch-feed!` binds
`*listener?*`), and a record write outside every operation (`LoggedRecords`).
## The logged record store
`attach` wraps a KB's record store in `LoggedRecords`, with a **watermark**: one above
every handle the store held when the log's generation began. The page cache can write
a record to disk before it writes the frame describing the operation that stored it,
so a crash can leave a record no durable frame accounts for. A replay finds every such
record at a handle at or above its own allocations, except a write to a handle below
the watermark — a deletion, a premise mark, a provenance change of a record the
generation began with. So before the first such write in an operation,
`LoggedRecords` fsyncs the operation's frame. An operation that only adds records
pays no fsync.
In **replay** mode (`attach-replaying`) the store allocates handles from the watermark
and checks each write against the record already stored at its handle: an equal record
is left as it is, a missing one is written, and a different one throws with
`::diverged` in its ex-data.
## Generations and replay
The file's first frame is a header naming the log's **generation**. `rotate!` starts
a generation: it truncates the file to a new header and clears the unusable mark.
`replay!` runs each operation frame through the installed dispatch with `*replaying?*`
bound, so `run-op` appends nothing and binds the frame's inputs, and with the change
feed off, since listeners belong to the process that made the writes. An operation
that refused when it was made refuses again, and replay goes on to the next frame.
`vaelii.impl.seal` decides which generation a restore replays and what a seal writes.
## Durability
One file, `<dir>/oplog/ops.log`, of length-prefixed nippy frames in
`vaelii.impl.disk.files`' format. The durability daemon fsyncs it on its tick and
closes it on shutdown. An open truncates a torn trailing frame, as the record logs'
opens do.True for the extent of a recorded operation, so a write nested inside it appends no frame of its own.
True for the extent of a recorded operation, so a write nested inside it appends no frame of its own.
True while a change-feed listener runs. A write made then marks the log unusable.
True while a change-feed listener runs. A write made then marks the log unusable.
The inputs map of the frame replay! is running.
The inputs map of the frame `replay!` is running.
True while replay! runs frames: run-op appends nothing and binds
*replay-inputs*.
True while `replay!` runs frames: `run-op` appends nothing and binds `*replay-inputs*`.
(attach kb log)kb recording its public writes into log: :oplog set, and the record store
wrapped in LoggedRecords whose watermark is one above a handle next-id allocates
now, so every record the store already held sits below it.
`kb` recording its public writes into `log`: `:oplog` set, and the record store wrapped in `LoggedRecords` whose watermark is one above a handle `next-id` allocates now, so every record the store already held sits below it.
(attach-replaying kb log watermark)kb attached to log in replay mode, allocating handles from watermark.
`kb` attached to `log` in replay mode, allocating handles from `watermark`.
(close-log! {:keys [raf lock reg]})Fsync and close log, and withdraw its durability registration.
Fsync and close `log`, and withdraw its durability registration.
(finish-replay! kb)Switch kb's logged record store from replay mode to normal mode.
Switch `kb`'s logged record store from replay mode to normal mode.
The frame layout's number, written into the log's header. A log of any other number is unusable.
The frame layout's number, written into the log's header. A log of any other number is unusable.
(generation log)The generation log holds.
The generation `log` holds.
(inner-records store)The record store under a LoggedRecords, or store itself.
The record store under a `LoggedRecords`, or `store` itself.
(install-dispatch! m)Install m, operation name -> function of the KB and the frame's arguments.
Install `m`, operation name -> function of the KB and the frame's arguments.
(mark-unusable! {:keys [raf lock state path]} reason)Record that log cannot be replayed, for reason. The first reason is the one kept.
Record that `log` cannot be replayed, for `reason`. The first reason is the one kept.
(open-log dir generation)Open the operation log under dir and register it with the durability daemon. A torn
trailing frame is truncated. A log with no header — a new one, or one a crash emptied
— starts at generation.
Open the operation log under `dir` and register it with the durability daemon. A torn trailing frame is truncated. A log with no header — a new one, or one a crash emptied — starts at `generation`.
(read-frames {:keys [raf lock]})The frames of log after its header, in append order, :unusable marks included.
The frames of `log` after its header, in append order, `:unusable` marks included.
(replay! kb frames)Run each operation frame of frames against kb (the namespace docstring,
"Generations and replay"). Throws with ::diverged in its ex-data when a replayed
write differs from the stored record, or a frame names an operation nothing installed.
Run each operation frame of `frames` against `kb` (the namespace docstring, "Generations and replay"). Throws with `::diverged` in its ex-data when a replayed write differs from the stored record, or a frame names an operation nothing installed.
(replay-next store)The next handle a replay-mode LoggedRecords would issue.
The next handle a replay-mode `LoggedRecords` would issue.
(request-seal! log)Ask for a seal when the running operation returns.
Ask for a seal when the running operation returns.
(rotate! {:keys [raf lock state]} gen)Start log again at generation gen: truncate the file to a header naming gen,
fsync it, and reset the state, the unusable mark included.
Start `log` again at generation `gen`: truncate the file to a header naming `gen`, fsync it, and reset the state, the unusable mark included.
(run-op kb op class args inputs-fn f)Run f as write operation op of class class on kb over args.
inputs-fn returns the inputs map a frame carries; f takes an inputs map and runs
the write with those inputs bound, or under the bindings in force when handed nil.
On a KB with no log, and nested inside a recorded operation, f runs on nil and
nothing is appended; a nested write made while *listener?* is true marks the log
unusable. Under replay!, f runs on the frame's inputs. When a top-level
operation returns, a requested seal runs.
Run `f` as write operation `op` of class `class` on `kb` over `args`. `inputs-fn` returns the inputs map a frame carries; `f` takes an inputs map and runs the write with those inputs bound, or under the bindings in force when handed nil. On a KB with no log, and nested inside a recorded operation, `f` runs on nil and nothing is appended; a nested write made while `*listener?*` is true marks the log unusable. Under `replay!`, `f` runs on the frame's inputs. When a top-level operation returns, a requested seal runs.
(set-seal-fn! log f)Install f, a function of the logged KB, as what run-op calls when a seal is due.
Install `f`, a function of the logged KB, as what `run-op` calls when a seal is due.
(set-watermark! store w)Move a LoggedRecords' watermark to w.
Move a `LoggedRecords`' watermark to `w`.
(unusable log)Why log cannot be replayed, or nil.
Why `log` cannot be replayed, or nil.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |