Block storage API. Functions which may cause IO to occur are marked with bangs.
For example (read! "foo")
doesn't have side-effects, but (read! some-input-stream)
will consume bytes from the stream.
When blocks are returned from a block store, they may include 'stat' metadata about the blocks, including:
:source
resource location for the block content:stored-at
time block was added to the storeBlock storage API. Functions which may cause IO to occur are marked with bangs. For example `(read! "foo")` doesn't have side-effects, but `(read! some-input-stream)` will consume bytes from the stream. When blocks are returned from a block store, they may include 'stat' metadata about the blocks, including: - `:source` resource location for the block content - `:stored-at` time block was added to the store
Block type and constructor functions.
Blocks have two primary attributes, :id
and :size
. The block identifier
is a Multihash
with the digest identifying the content. The size is the
number of bytes in the block content.
Internally, blocks either have in-memory content holding the data, or a reader function which returns new input streams for the block data. A block with in-memory content is a loaded block, while a block with a reader function is a lazy block.
A block's id, size, content, and reader cannot be changed after construction, so clients can be relatively certain that the block's id is valid. Blocks may have additional attributes associated with them and support metadata, similar to records.
Block type and constructor functions. Blocks have two primary attributes, `:id` and `:size`. The block identifier is a `Multihash` with the digest identifying the content. The size is the number of bytes in the block content. Internally, blocks either have in-memory content holding the data, or a reader function which returns new input streams for the block data. A block with in-memory content is a _loaded block_, while a block with a reader function is a _lazy block_. A block's id, size, content, and reader cannot be changed after construction, so clients can be relatively certain that the block's id is valid. Blocks _may_ have additional attributes associated with them and support metadata, similar to records.
Instrumentation for block stores to measure data flows, call latencies, and other metrics.
The logic in this namespace is built around the notion of a metric event and
an associated recording function on the store which the events are passed
to. Each event has at least a namespaced :type
keyword, a :label
associated with the store, and a numeric :value
.
Events may contain other information like the block id or method name as well, and it is up to the receiver to interpret them.
Instrumentation for block stores to measure data flows, call latencies, and other metrics. The logic in this namespace is built around the notion of a _metric event_ and an associated _recording function_ on the store which the events are passed to. Each event has at least a namespaced `:type` keyword, a `:label` associated with the store, and a numeric `:value`. Events may contain other information like the block id or method name as well, and it is up to the receiver to interpret them.
Logical block storage which buffers new blocks being written to a backing store. Reads return a unified view of the existing and buffered blocks. The buffer can be flushed to write all the new blocks to the backing store.
Logical block storage which buffers new blocks being written to a backing store. Reads return a unified view of the existing and buffered blocks. The buffer can be _flushed_ to write all the new blocks to the backing store.
Cache stores provide logical block storage backed by two other stores, a primary store and a cache. Blocks are added to the cache on reads and writes, and evicted with a least-recently-used strategy to keep the cache under a certain total size. Operations on this store will prefer to look up blocks in the cache, and fall back to the primary store when not available.
Because the caching logic runs locally, the backing cache storage should not be shared among multiple concurrent processes.
Cache stores provide logical block storage backed by two other stores, a _primary store_ and a _cache_. Blocks are added to the cache on reads and writes, and evicted with a least-recently-used strategy to keep the cache under a certain total size. Operations on this store will prefer to look up blocks in the cache, and fall back to the primary store when not available. Because the caching logic runs locally, the backing cache storage should not be shared among multiple concurrent processes.
Block storage backed by files in nested directories. Each block is stored in
a separate file. File block stores may be constructed using the
file://<path-to-root>
URI form.
In many filesystems, performance degrades as the number of files in a directory grows. In order to reduce this impact and make navigating the blocks a bit more efficient, block files are stored in directories under the store root. All path elements are lower-case hex-encoded bytes from the multihash.
The directories under the root consist of the first four bytes of the multihashes of the blocks stored in them. Within each directory, blocks are stored in files whose names consist of the rest of their digests.
Thus, a block containing the content foobar
would have the sha1 digest
97df3501149...
and be stored under the root directory at:
root/111497df/35011497df3588b5a3...
This implementation tries to match the IPFS fs-repo behavior so that the on-disk representations remain compatible.
Block storage backed by files in nested directories. Each block is stored in a separate file. File block stores may be constructed using the `file://<path-to-root>` URI form. In many filesystems, performance degrades as the number of files in a directory grows. In order to reduce this impact and make navigating the blocks a bit more efficient, block files are stored in directories under the store root. All path elements are lower-case hex-encoded bytes from the multihash. The directories under the root consist of the first four bytes of the multihashes of the blocks stored in them. Within each directory, blocks are stored in files whose names consist of the rest of their digests. Thus, a block containing the content `foobar` would have the sha1 digest `97df3501149...` and be stored under the root directory at: `root/111497df/35011497df3588b5a3...` This implementation tries to match the IPFS fs-repo behavior so that the on-disk representations remain compatible.
Block storage backed by a map in an atom. Blocks put into this store will be
passed to load!
to ensure the content resides in memory. Memory block stores
may be constructed usin the mem:-
URI form.
This store is most suitable for testing, caches, and other situations which call for a non-persistent block store.
Block storage backed by a map in an atom. Blocks put into this store will be passed to `load!` to ensure the content resides in memory. Memory block stores may be constructed usin the `mem:-` URI form. This store is most suitable for testing, caches, and other situations which call for a non-persistent block store.
Logical block storage which writes to multiple backing stores to ensure durability. Lookups will try the backing stores in order to find blocks.
Logical block storage which writes to multiple backing stores to ensure durability. Lookups will try the backing stores in order to find blocks.
Suite of tests to verify that a given block store implementation conforms to the spec.
Suite of tests to verify that a given block store implementation conforms to the spec.
A 'summary' represents a collection of blocks, including certain statistics and an imprecise membership test in the form of a bloom filter. These are useful for returning from certain operations to represent the set of blocks acted upon.
A 'summary' represents a collection of blocks, including certain statistics and an imprecise membership test in the form of a bloom filter. These are useful for returning from certain operations to represent the set of blocks acted upon.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close