Durability management for the disk backend. Each disk store/kv registers itself here on open; one daemon fsyncs every registrant on a tick (default 3 s), and one JVM shutdown hook closes them all on exit. Without it, a crash between manual fsyncs loses everything since the last one.
Registration is capability-based — callers hand in {:fsync :close :label} (plus an
optional {:compact :dead-ratio} for background compaction), so this namespace does
not depend on the stores it drives (which would cycle).
Config (system properties): vaelii.disk.sync-ms (tick interval, 0 disables the
daemon), vaelii.disk.auto-compact, vaelii.disk.compact-dead-ratio (default 0.5),
vaelii.disk.compact-min-interval-ms (default 300000). Every one is read through
vaelii.impl.config, which owns their domains and refuses a value outside one —
and the two the tick reads are why config/check! runs at the open: a refusal here
lands inside fsync-all's catch Throwable below, which logs a class name every
three seconds and leaves auto-compaction dead.
Durability management for the disk backend. Each disk store/kv registers itself
here on open; one daemon fsyncs every registrant on a tick (default 3 s), and one
JVM shutdown hook closes them all on exit. Without it, a crash between manual
fsyncs loses everything since the last one.
Registration is capability-based — callers hand in `{:fsync :close :label}` (plus an
optional `{:compact :dead-ratio}` for background compaction), so this namespace does
not depend on the stores it drives (which would cycle).
Config (system properties): `vaelii.disk.sync-ms` (tick interval, 0 disables the
daemon), `vaelii.disk.auto-compact`, `vaelii.disk.compact-dead-ratio` (default 0.5),
`vaelii.disk.compact-min-interval-ms` (default 300000). Every one is read through
`vaelii.impl.config`, which owns their domains and refuses a value outside one —
and the two the tick reads are why `config/check!` runs at the open: a refusal *here*
lands inside `fsync-all`'s `catch Throwable` below, which logs a class name every
three seconds and leaves auto-compaction dead.(auto-compact?)Is background/opportunistic compaction enabled (vaelii.disk.auto-compact)? Public
because the close path consults the same switch the tick does — one knob, not two.
Is background/opportunistic compaction enabled (`vaelii.disk.auto-compact`)? Public because the close path consults the same switch the tick does — one knob, not two.
(await-compaction-quiescent! ids)(await-compaction-quiescent! ids timeout-ms)Block until none of ids has an auto-compaction in flight. Returns true when they
are quiet, false on the timeout (logged, naming the ids that would not settle).
What it is for. The compaction executor runs a rewrite on its own thread, and the
record store's rewrite phase deliberately holds no lock while it does — so a caller
that is about to hand the directory to somebody else (backend/close-dir!, before
lock/release!) cannot learn from any lock that the rewrite is done. It has to ask
here. Pair it with the store's own abort-compaction!: the abort is what makes the
wait short, this is what makes it correct.
Waiting on compaction-in-flight rather than on the executor itself, because the
executor is process-wide and one directory's close has no business joining another
directory's rewrite. The set is read under the same monitor the notify takes, so a
task that finishes between the read and the wait cannot be missed.
It gives up rather than throwing, on the timeout and on an interrupt alike, and
both say so. The caller is a close, and a close that threw here would unwind before
lock/release! — leaving the directory marked held for the life of the process over a
wait that did not finish, which is a worse version of the thing this exists to
prevent. A false says the join is not a guarantee this time; the close carries on and
hands the directory over, and the log line is what an operator reads.
Block until none of `ids` has an auto-compaction in flight. Returns true when they are quiet, false on the timeout (logged, naming the ids that would not settle). **What it is for.** The compaction executor runs a rewrite on its own thread, and the record store's rewrite phase deliberately holds no lock while it does — so a caller that is about to hand the directory to somebody else (`backend/close-dir!`, before `lock/release!`) cannot learn from any lock that the rewrite is done. It has to ask here. Pair it with the store's own `abort-compaction!`: the abort is what makes the wait short, this is what makes it correct. Waiting on `compaction-in-flight` rather than on the executor itself, because the executor is process-wide and one directory's close has no business joining another directory's rewrite. The set is read under the same monitor the notify takes, so a task that finishes between the read and the wait cannot be missed. **It gives up rather than throwing**, on the timeout and on an interrupt alike, and both say so. The caller is a close, and a close that threw here would unwind before `lock/release!` — leaving the directory marked held for the life of the process over a wait that did not finish, which is a worse version of the thing this exists to prevent. A false says the join is not a guarantee this time; the close carries on and hands the directory over, and the log line is what an operator reads.
(call-with-compaction-paused thunk)Run thunk with background auto-compaction paused, resuming on the way out (even on
throw). A no-op in effect when no disk backend is registered.
Run `thunk` with background auto-compaction paused, resuming on the way out (even on throw). A no-op in effect when no disk backend is registered.
(compact-dead-ratio)The dead-ratio a log must reach to be worth compacting
(vaelii.disk.compact-dead-ratio, default 0.5). Public for the same reason.
The dead-ratio a log must reach to be worth compacting (`vaelii.disk.compact-dead-ratio`, default 0.5). Public for the same reason.
(deregister! id)Remove a registered backend. Idempotent.
Remove a registered backend. Idempotent.
(pause-compaction!)Suspend the daemon's background auto-compaction across every registered backend.
For a bulk load, whose monotonic delta accumulation trips the dead-ratio trigger
repeatedly and would rewrite a growing multi-GB index mid-load — stalling the writer
on the backend lock each time — pause for the load's duration and let the next tick
compact once afterwards. Idempotent; pair with resume-compaction!, or wrap the load
in call-with-compaction-paused. Only the daemon's automatic firing is gated: a
manual compact! and the dead-ratio bookkeeping are untouched.
Suspend the daemon's background auto-compaction across every registered backend. For a bulk load, whose monotonic delta accumulation trips the dead-ratio trigger repeatedly and would rewrite a growing multi-GB index mid-load — stalling the writer on the backend lock each time — pause for the load's duration and let the next tick compact once afterwards. Idempotent; pair with `resume-compaction!`, or wrap the load in `call-with-compaction-paused`. Only the daemon's automatic firing is gated: a manual `compact!` and the dead-ratio bookkeeping are untouched.
(register! {:keys [fsync close label phase] :as entry})Register a disk backend with the durability manager. Entry keys: :fsync (fn of one
argument, required — the tick hands it an options map, which it passes empty), :close
(fn [], required), :label (string, required), optionally :compact (fn []) +
:dead-ratio (fn [] → double) for background compaction, and optionally :phase
(close-phases, :store by default) for shutdown ordering. Returns an id for
deregister!; starts the scheduler and installs the shutdown hook on first
registration, both under lifecycle so concurrent first registrations install one
apiece rather than one each.
The shape check is an ex-info rather than an assert because clojure.core/assert
is elidable: with *assert* false a registrant with no :close would register
cleanly and then be silently skipped on shutdown, which is the whole of what this
namespace does for it.
Register a disk backend with the durability manager. Entry keys: `:fsync` (fn of one argument, required — the tick hands it an options map, which it passes empty), `:close` (fn `[]`, required), `:label` (string, required), optionally `:compact` (fn `[]`) + `:dead-ratio` (fn `[]` → double) for background compaction, and optionally `:phase` (`close-phases`, `:store` by default) for shutdown ordering. Returns an id for `deregister!`; starts the scheduler and installs the shutdown hook on first registration, both under `lifecycle` so concurrent first registrations install one apiece rather than one each. The shape check is an `ex-info` rather than an `assert` because `clojure.core/assert` is **elidable**: with `*assert*` false a registrant with no `:close` would register cleanly and then be silently skipped on shutdown, which is the whole of what this namespace does for it.
(resume-compaction!)Re-enable background auto-compaction paused by pause-compaction!. The next flush
tick compacts any backend whose dead ratio has crossed the threshold.
Re-enable background auto-compaction paused by `pause-compaction!`. The next flush tick compacts any backend whose dead ratio has crossed the threshold.
(stop!)Stop the schedulers (REPL/test teardown). Leaves the shutdown hook installed. Under
the same monitor the starts take, so a register! racing this either installs before it
or re-installs after it, never half-way through it.
The stopped flag is the half a reset! to nil cannot do: a tick already inside
submit-compaction! reads the nil'd atom, and without the flag it builds a
replacement executor — one nothing holds a reference to, so stop! returned having
stopped nothing. Cleared by the next register!, which is what a REPL teardown
followed by a fresh open looks like.
Stop the schedulers (REPL/test teardown). Leaves the shutdown hook installed. Under the same monitor the starts take, so a `register!` racing this either installs before it or re-installs after it, never half-way through it. The stopped flag is the half a `reset!` to nil cannot do: a tick already inside `submit-compaction!` reads the nil'd atom, and without the flag it builds a *replacement* executor — one nothing holds a reference to, so `stop!` returned having stopped nothing. Cleared by the next `register!`, which is what a REPL teardown followed by a fresh open looks like.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |