All notable changes to this project will be documented in this file, which follows the conventions of keepachangelog.com. This project adheres to Semantic Versioning.
...
blocks.data/bounded-input-stream public as a utility for implementing
block stores.multiformats.hash/parse method.blocks-test project to blocks-tests to better reflect the
test namespace.This is a major release with a rewritten storage interface. Much of the library's API has changed slightly - the core concepts are the same, but now storage interactions are represented as asynchronous processes using manifold. Please read the notes here carefully!
The first major change is that block values are no longer open maps - they behave more like opaque types now. Mixing in extra attributes led to some confusing usage in practice; one example was the way extra attributes affected equality, meaning you could not test blocks directly for content matches. Blocks do still support metadata, so any additional information can still be associated that way.
Second, this version upgrades from the mvxcvi/multihash library to the unified
mvxcvi/multiformats code. This changes the type of a block :id from
multihash.core.Multihash to multiformats.hash.Multihash, but otherwise the
identifiers have the same semantics and should behave the same way they did
before.
Blocks also have a new first-class attribute :stored-at which is a
java.time.Instant reflecting the time they were persisted. This does not
affect block equality or hashing, but is generally useful for auditing. It
does impact sorting, so that earlier copies of the same block sort before
older ones.
Finally, block content is no longer represented by separate reader and
content fields on the block. Now the content field contains an
implementation of the new blocks.data/ContentReader protocol. This is
implemented for the current PersistentBytes values and the "reader function"
approach for lazy blocks. The protocol allows block stores to provide an
efficient mechanism for reading a sub-range of the block content, and will be
useful for any future customizations.
The major change in this version of the library is that all block store methods
are now asynchronous. The stat, get, store!, delete!, get-batch!,
put-batch!, delete-batch!, scan, erase!, and sync functions now return
manifold deferred values instead of blocking.
Similarly, the list store method now returns a manifold stream instead of a
lazy sequence. An asynchronous process places blocks on this stream for
consumption - previously, this was simple stat metadata. If an error occurs, the
store should place the exception on the stream and close it. Existing consumers
can use the list-seq wrapper, which returns a lazy sequence consuming from
this stream and behaves similarly to the old list method.
The list and list-seq query parameters now also accept a :before hex
string (in addition to the current :after) to halt enumeration at a certain
point.
The blocks.store/BatchStore protocol has been removed. It was never used in
practice and few backends could ensure atomicity. Instead, the batch methods are
now wrappers around asynchronous behavior over the normal store methods.
The library now includes a blocks.meter namespace which provides a common
framework for instrumenting block stores and collecting metrics. Users can opt
into metrics collection by setting a :blocks.meter/recorder function on each
store they want to instrument. This function will be called with the store
record and metric events, which include method elapsed times, traffic into and
out of the store, and counts of blocks enumerated by list streams.
In addition to the notes for each store below, note that external block store libraries like blocks-s3 will not be compatible with this version of the library until they upgrade!
primary instead of store.primary and buffer stores
are present.clear! and flush! now return deferred values.predicate function which can return false
to indicate that a block should be stored directly in the primary store
instead of being buffered.:stored-at attributes so that younger blocks are preferred.reap! now returns a deferred value.max-block-size field, the caching store now supports an
arbitrary predicate function, which can return false to indicate that a
block should not be cached.replicas instead of store-keys.The file block store has seen the most significant changes. Previously, blocks
were stored in subdirectories under the store root, like
$ROOT/1220abcd/0123..., with no additional metadata. Now, file stores maintain
a more sophisticated structure under the root. Block directories are now in
$ROOT/blocks/, and a $ROOT/meta.properties file contains versioning
information to make future extensibility possible. When the store starts, it
will try to detect a v0 layout; if :auto-migrate? is truthy on the store, it
will upgrade it to v1, otherwise it will throw an exception.
Another change is that blocks are now written to a temporary file in
$ROOT/landing/ before being atomically renamed to their final location. This
keeps other clients from seeing partially-written blocks that are still being
stored - something that would have been difficult with the prior layout.
A few other things changed or were added:
blocks.core/loaded? which is the complement of lazy?.blocks.core/open now accepts a map as a second argument instead of a
three-arity (open block start end). Instead this would now be
(open block {:start start, :end end})blocks.core/validate! now returns true instead of nil on success.blocks.summary aggregates no longer contain bloom filters; these didn't
seem to be used in practice, and clients which want that behavior can
reimplement it without much difficulty.blocks.store.tests have moved to a separate subproject
mvxcvi/blocks-test to simplify usage by store implementations.This release upgrades the library to Clojure 1.9.0.
Finally seems like time for a 1.0 release. One very minor breaking change.
blocks.core/lazy?.:stored-at metadata on blocks is now returned as a
java.time.Instant instead of a java.util.Date.MemoryBlockStore uses a ref internally instead of an atom.test.carly.IDeref as a way to get their internal
content.PersistentBytes has a toByteArray method to return a copy of the byte data
as a raw array.This release has a couple of breaking changes, detailed below.
PersistentBytes values support comparison using lexical sorting rules.blocks.core/->store initializer function to create block stores from URI
configuration strings.blocks.core/scan function to produce a summary of the blocks contained in
the store.blocks.core/sync! function to copy blocks between stores.ErasableStore protocol for block stores which support efficient or atomic
data removal. There's a matching blocks.core/erase!! function using it,
which falls back to deleting the blocks in the store individually.blocks.store.util namespace merged into blocks.store. This mainly impacts
store implementers.EnumerableStore protocol and enumerate method. No usages have
come up requiring it and it's easy to replace in the non-optimized case.BlockStore.file-block-store.IPending, because it is not appropriate to treat
immutable values as asynchronous references.blocks.store/initialize for constructing block stores from a
URI string. The method is dispatched by URI scheme.blocks.data.conversions, which defined conversion
paths for the byte-streams library.blocks.data/clean-block.mvxcvi/multihash to 2.0.0.put! in them.blocks.store.tests now build generative sequences of
operations and apply them to the store under test.nil instead of an empty block.block/put! is actually a block.block/put! instead of requiring stores to do it.false when deleting a block which is not
contained in the store.PersistentBytes equality to include primitive byte arrays and
ByteBuffer objects which have identical content.block/store! will no longer try to store empty files.block/open to read a sub-range of the content in a
block by specifying starting and ending bytes.
#3FileBlockStore to FileStore.blocks.store namespace, with wrappers in
blocks.core.validate! now checks the size of lazy blocks by using a counting input
stream wrapper.blocks.store.cache namespace with logical caching block store
implementation.random-bytes and random-hex now generate fixed-width data.put!, delete!, and erase! to
prevent concurrent modifications.select-stats moved from core to util namespace.:origin block stat to :source.read-block for consistency.put! retains extra attributes and metadata on the block argument in the
returned block.stat and get on non-existent
blocks and put! merging.Lots of high-level library changes! blocks.data.Block is now a custom type to
protect immutable fields like :id and :size and support the IPending
interface.
blocks.store.tests namespace.BlockStore methods enumerate and get* changed to -list and -get,
respectively.list now returns a sequence of block stats, rather than just multihashes.get and put! add stat information as metadata.Initial project release.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |