Test instrumentation: bind to an atom and read-kv-pair! counts every entry it
deserializes, so a test can prove an index-directed read touches only its block
rather than scanning the file. nil in production -- a nil check, no atom traffic
on the read path.
Test instrumentation: bind to an atom and `read-kv-pair!` counts every entry it deserializes, so a test can prove an index-directed read touches only its block rather than scanning the file. nil in production -- a nil check, no atom traffic on the read path.
(append-entry! out-stream [ikey data])Write one entry: key segment (seq ++ user_key) then value segment (or a
length-0 tombstone). Used by both the WAL and SSTable writers.
Write one entry: key segment (`seq ++ user_key`) then value segment (or a length-0 tombstone). Used by both the WAL and SSTable writers.
(crc32 data)Return the unsigned CRC32 value for data.
File formats choose how many bytes encode the value: regular framed segments store it as a long, while the manifest's fixed header stores it as an int.
Return the unsigned CRC32 value for `data`. File formats choose how many bytes encode the value: regular framed segments store it as a long, while the manifest's fixed header stores it as an int.
(decode-index payload)Parse an index payload into a vector of [first-user_key block-offset], in user_key order (ready for binary search).
Parse an index payload into a vector of [first-user_key block-offset], in user_key order (ready for binary search).
(decode-wal-record payload)Parse a WAL record payload into [seq entries], where each entry is
[ikey data] with ikey's seq set to the record's seq.
Parse a WAL record payload into `[seq entries]`, where each entry is `[ikey data]` with `ikey`'s seq set to the record's seq.
(encode-index blocks)Serialize the sparse index: block-count, then per block keylen ++ first-user_key ++ block-start-offset.
Serialize the sparse index: block-count, then per block keylen ++ first-user_key ++ block-start-offset.
(encode-wal-record seq entries)Serialize one tx (its seq and folded [ikey data] entries) into the record
payload bytes. The caller frames it with write-bytes! (length + CRC).
Serialize one tx (its `seq` and folded `[ikey data]` entries) into the record payload bytes. The caller frames it with `write-bytes!` (length + CRC).
(entry-size-on-disk ikey data)Exact number of bytes append-entry! writes for one entry: the key segment
(len + seq + user_key + CRC) plus the value segment (len + value + CRC, or a
bare length-0 long for a tombstone). Authoritative -- the SSTable writer uses it
to cut index blocks and compaction uses it to size output tables.
Exact number of bytes `append-entry!` writes for one entry: the key segment (len + seq + user_key + CRC) plus the value segment (len + value + CRC, or a bare length-0 long for a tombstone). Authoritative -- the SSTable writer uses it to cut index blocks and compaction uses it to size output tables.
(floor-block-offset index target-key)Byte offset of the block a read for target-key must start at (see
floor-block-idx). Offset 1 -- just past the format byte -- when there is no
index.
Byte offset of the block a read for `target-key` must start at (see `floor-block-idx`). Offset 1 -- just past the format byte -- when there is no index.
(floor-block-span index target-key index-start)[start end) for a read of target-key: the floor block's offset and where
that block ends (the next block's offset, or the end of the entries region).
Sizing the first channel read to the block avoids pulling in bytes the read will
never look at.
`[start end)` for a read of `target-key`: the floor block's offset and where that block ends (the next block's offset, or the end of the entries region). Sizing the first channel read to the block avoids pulling in bytes the read will never look at.
(read-all-entries file-path)Every [ikey data] entry of an SSTable, in file order (compaction reads whole tables). Bounded by the entries region -- reading to EOF would run into the index/trailer footer and misparse it as entries. Opens its own channel: a compaction reads each input exactly once, so it bypasses the read cache.
Every [ikey data] entry of an SSTable, in file order (compaction reads whole tables). Bounded by the entries region -- reading to EOF would run into the index/trailer footer and misparse it as entries. Opens its own channel: a compaction reads each input exactly once, so it bypasses the read cache.
(read-bytes! in-stream file-path)Read a data segment that is required to be present and valid, e.g. a field of a fixed-format file (SSTable info). Throws on any EOF/truncation/corruption.
Read a data segment that is required to be present and valid, e.g. a field of a fixed-format file (SSTable info). Throws on any EOF/truncation/corruption.
(read-data! in-stream)Read one length-prefixed, CRC-checked data segment, classifying the outcome so callers can distinguish the four error classes. Returns one of:
:eof - a clean end at a record boundary (nothing left to read) :truncated - an incomplete entry: EOF was reached partway through the length prefix, the data, or the CRC. Normal only at a WAL tail (a crash before the write's fsync completed). :tombstone - a deletion marker (segment length 0) :corrupt - the whole segment was read but its CRC did not match, or the length prefix is nonsensical (mid-file corruption) {:ok bytes} - a valid data segment
Read one length-prefixed, CRC-checked data segment, classifying the outcome
so callers can distinguish the four error classes. Returns one of:
:eof - a clean end at a record boundary (nothing left to read)
:truncated - an incomplete entry: EOF was reached partway through the
length prefix, the data, or the CRC. Normal only at a WAL
tail (a crash before the write's fsync completed).
:tombstone - a deletion marker (segment length 0)
:corrupt - the whole segment was read but its CRC did not match, or the
length prefix is nonsensical (mid-file corruption)
{:ok bytes} - a valid data segment(read-footer! file-path)Read the trailer from the file end, then the index region. Returns
{:index-start <byte offset where entries end> :index [[user_key offset]...]}.
Raises on a corrupt/truncated footer or an unsupported format version.
Read the trailer from the file end, then the index region. Returns
`{:index-start <byte offset where entries end> :index [[user_key offset]...]}`.
Raises on a corrupt/truncated footer or an unsupported format version.(read-format-byte! in-stream file-path)Consume and validate the leading SSTable format-version byte.
Consume and validate the leading SSTable format-version byte.
(read-kv-pair! in-stream)Read one entry (key segment then value segment). Returns:
:eof - a clean end at a record boundary
:truncated - an incomplete entry at the tail
:corrupt - a CRC mismatch (mid-file corruption)
[ikey data] - a complete entry; ikey is an InternalKey, data a Data
Read one entry (key segment then value segment). Returns: :eof - a clean end at a record boundary :truncated - an incomplete entry at the tail :corrupt - a CRC mismatch (mid-file corruption) [ikey data] - a complete entry; `ikey` is an InternalKey, `data` a `Data`
(read-latest-seq ch file-path target-key {:keys [index index-start]})The seq of the newest version of target-key in an SSTable, or nil if absent.
Entries are user_key-asc / seq-desc, so the first entry with user_key == target
is the newest. Used by commit-time write-write conflict detection.
The seq of the *newest* version of `target-key` in an SSTable, or nil if absent. Entries are user_key-asc / seq-desc, so the first entry with user_key == target is the newest. Used by commit-time write-write conflict detection.
(read-value ch file-path target-key snapshot-seq {:keys [index index-start]})Point read: the newest version of target-key (a user_key) with seq <=
snapshot-seq, or nil. Entries are user_key-asc / seq-desc, so the first entry
with user_key == target and seq <= snapshot is the answer.
footer is the table's in-memory {:index :index-start}: the read starts at the
floor block for target-key rather than the file start, and stops where the
entries end. It keeps scanning past block ends until the user_key exceeds the
target, so a user_key whose versions straddle a block boundary is still found in
full. Bytes come off the cached channel via absolute positional reads.
Point read: the newest version of `target-key` (a user_key) with seq <=
`snapshot-seq`, or nil. Entries are user_key-asc / seq-desc, so the first entry
with user_key == target and seq <= snapshot is the answer.
`footer` is the table's in-memory `{:index :index-start}`: the read starts at the
floor block for `target-key` rather than the file start, and stops where the
entries end. It keeps scanning past block ends until the user_key exceeds the
target, so a user_key whose versions straddle a block boundary is still found in
full. Bytes come off the cached channel via absolute positional reads.(read-wal-record! in-stream)Read one framed WAL record. Returns :eof / :truncated / :corrupt (as
read-data! classifies the frame) or [seq entries] for a complete tx.
Read one framed WAL record. Returns `:eof` / `:truncated` / `:corrupt` (as `read-data!` classifies the frame) or `[seq entries]` for a complete tx.
(scan-pairs ch
file-path
from-key
to-key
snapshot-seq
{:keys [index index-start]})Range read: for each user_key in [from-key, to-key), its newest version with
seq <= snapshot-seq, as [user_key data] (tombstones included; callers filter).
Entries are user_key-asc / seq-desc. Starts at the block covering from-key.
Range read: for each user_key in [from-key, to-key), its newest version with seq <= `snapshot-seq`, as [user_key data] (tombstones included; callers filter). Entries are user_key-asc / seq-desc. Starts at the block covering `from-key`.
(write-footer! out-stream blocks index-start)Append the index region then the fixed-size trailer. index-start is the byte
offset just past the last entry (i.e. where the index region begins).
Append the index region then the fixed-size trailer. `index-start` is the byte offset just past the last entry (i.e. where the index region begins).
(write-format-byte! out-stream)Write the one-byte SSTable format version at the start of a file.
Write the one-byte SSTable format version at the start of a file.
(write-wal-record! out-stream seq entries)Append one framed WAL record (one tx) to a WAL output stream.
Append one framed WAL record (one tx) to a WAL output stream.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |