A live-handle roster: what sentex-ids / justification-ids / premise-ids hand
back, for a store big enough that the shape matters.
The three enumerations answer a java.util.Set of handles, and every store the engine
ships answers a PersistentHashSet<Long> because that is what its own state already
is. At the scale a server-backed store exists for, that shape is the cost: measured
over contiguous handles, a PersistentHashSet<Long> retains 48–75 bytes per handle
(the hash trie's fill varies with cardinality), so 4.5–7.0 GB at 100M — and
rebuild-tms holds the sentex roster while it walks the premises and the
justifications.
Handles are minted in assertion order (next-id), so a roster is a near-contiguous run
of longs with holes where records were deleted — the one shape a compressed bitmap is
built for. Over the same handles with a tenth of them punched out, a Roaring64Bitmap
retains 0.13–0.26 bytes per handle, and less than that on an unbroken run, where
the whole roster is a few run containers. It answers contains? in less time than
the hash set, not more (vaelii.impl.protocols, the enumeration contract).
Not an IPersistentSet. conj, disj and clojure.set need one, so a caller wanting
those converts with (set roster) — which is the 4.5 GB, paid at the call site that
asked for it rather than by every store on every enumeration. What the seam promises is
membership, iteration, cardinality and ordering, which is what every caller in the
engine uses.
Immutable once built, so concurrent readers need no coordination — the one property the bitmap has to have here, since a store's readers enumerate while its writer writes.
A **live-handle roster**: what `sentex-ids` / `justification-ids` / `premise-ids` hand back, for a store big enough that the shape matters. The three enumerations answer a `java.util.Set` of handles, and every store the engine ships answers a `PersistentHashSet<Long>` because that is what its own state already is. At the scale a server-backed store exists for, that shape *is* the cost: measured over contiguous handles, a `PersistentHashSet<Long>` retains **48–75 bytes per handle** (the hash trie's fill varies with cardinality), so **4.5–7.0 GB at 100M** — and `rebuild-tms` holds the sentex roster while it walks the premises and the justifications. Handles are minted in assertion order (`next-id`), so a roster is a near-contiguous run of longs with holes where records were deleted — the one shape a compressed bitmap is built for. Over the same handles with a tenth of them punched out, a `Roaring64Bitmap` retains **0.13–0.26 bytes per handle**, and less than that on an unbroken run, where the whole roster is a few run containers. It answers `contains?` in *less* time than the hash set, not more (`vaelii.impl.protocols`, the enumeration contract). ## What this is not Not an `IPersistentSet`. `conj`, `disj` and `clojure.set` need one, so a caller wanting those converts with `(set roster)` — which is the 4.5 GB, paid at the call site that asked for it rather than by every store on every enumeration. What the seam promises is membership, iteration, cardinality and ordering, which is what every caller in the engine uses. Immutable once built, so concurrent readers need no coordination — the one property the bitmap has to have here, since a store's readers enumerate while its writer writes.
(collector)A mutable [add! finish] pair, for a caller with a row loop rather than a reducible —
a SQL store's cursor walk. add! takes one handle; finish returns the immutable
roster and is called once.
A mutable `[add! finish]` pair, for a caller with a row loop rather than a reducible — a SQL store's cursor walk. `add!` takes one handle; `finish` returns the immutable roster and is called once.
(roster ids)The handles in ids as a HandleRoster. ids is anything reducible — a seq, a
vector, an array — and need not arrive sorted.
The handles in `ids` as a `HandleRoster`. `ids` is anything reducible — a seq, a vector, an array — and need not arrive sorted.
(roster? x)Is x one of these? For a caller deciding whether (set x) would cost anything.
Is `x` one of these? For a caller deciding whether `(set x)` would cost anything.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |