Compaction: checkpoint the live state, prune the log it covers.
The RAM-bound ceiling (ADR-0005) is that a file store replays its whole log at boot — every correction a store ever absorbed costs boot RAM forever. Compaction bounds that replay to the live set: the current datoms as of one t go into a checkpoint SEGMENT (the protocol's segment half, exercised at last), and the log entries the checkpoint covers are pruned. Boot then replays checkpoint + tail.
The order is the whole correctness story. The checkpoint put is durable BEFORE any pruning starts, and replay starts after the checkpoint's base, so every crash window replays identically:
What pruning costs is history below the base: :as-of t for t < base raises :db/pruned-t, and history shows surviving datoms plus the full post-compaction story. Corrections before the base stay corrected — the checkpoint holds their outcome — but the trail to them is gone. That is the trade ADR-0008 records; compaction is therefore an explicit, opt-in call (kala.api/compact), never a background process.
Checkpoint datoms keep their original tx coordinates, so time-travel reads above the base are exact, and the checkpoint carries the user entity-id allocation counter explicitly — a live-state-only max could not recover the ids of entities whose every datom was retracted.
Compaction: checkpoint the live state, prune the log it covers.
The RAM-bound ceiling (ADR-0005) is that a file store replays its whole
log at boot — every correction a store ever absorbed costs boot RAM
forever. Compaction bounds that replay to the live set: the current
datoms as of one t go into a checkpoint SEGMENT (the protocol's segment
half, exercised at last), and the log entries the checkpoint covers are
pruned. Boot then replays checkpoint + tail.
The order is the whole correctness story. The checkpoint put is durable
BEFORE any pruning starts, and replay starts after the checkpoint's base,
so every crash window replays identically:
- crash before the put: nothing happened;
- crash after the put, before the prune: checkpoint + full log — replay
reads checkpoint, skips entries up to the base, ignores nothing new;
- crash mid-prune: leftover pre-base entries are skipped by range.
What pruning costs is history below the base: :as-of t for t < base
raises :db/pruned-t, and history shows surviving datoms plus the full
post-compaction story. Corrections before the base stay corrected — the
checkpoint holds their outcome — but the trail to them is gone. That is
the trade ADR-0008 records; compaction is therefore an explicit,
opt-in call (kala.api/compact), never a background process.
Checkpoint datoms keep their original tx coordinates, so time-travel
reads above the base are exact, and the checkpoint carries the user
entity-id allocation counter explicitly — a live-state-only max could
not recover the ids of entities whose every datom was retracted.(checkpoint db)The checkpoint document for a db value: its t, the user allocation counter, and the live datoms.
The checkpoint document for a db value: its t, the user allocation counter, and the live datoms.
(checkpoint-base storage)The t of storage's current checkpoint, or nil when the store has never been compacted.
The t of storage's current checkpoint, or nil when the store has never been compacted.
(compact! conn)Compact a peer connection's store: write the checkpoint segment for the connection's current t, then prune the log entries at or below it. Returns {:t the-base :datoms checkpointted :pruned pruned}.
Crash-safe by ordering (see the namespace): the put acks durable before the prune starts, and both halves are idempotent, so re-running after an interruption finishes the job. Concurrent transacts are safe to interleave — they land above the base and never touch the pruned range — and the connection is re-annotated atomically so later as-of reads know where history now begins.
Compact a peer connection's store: write the checkpoint segment for the
connection's current t, then prune the log entries at or below it.
Returns {:t the-base :datoms checkpointted :pruned pruned}.
Crash-safe by ordering (see the namespace): the put acks durable
before the prune starts, and both halves are idempotent, so re-running
after an interruption finishes the job. Concurrent transacts are safe
to interleave — they land above the base and never touch the pruned
range — and the connection is re-annotated atomically so later as-of
reads know where history now begins.(live-raw-datoms db)The db value's live state as raw storage tuples — [e attr-id v tx true] in EAVT order, attr ids raw, each datom's original tx kept.
The db value's live state as raw storage tuples — [e attr-id v tx true] in EAVT order, attr ids raw, each datom's original tx kept.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |