link:../src/crux/db.clj[role=include]
Crux is an unbundled database, where parts are pluggable and can be swapped out for alternative implementations. We are attempting to follow the Unix philosophy of each part doing one thing well.
This document walks through some of the high-level Crux components.
Crux contains an ObjectStore for storing and retreiving documents.
link:../src/crux/db.clj[role=include]
The main implementation is KvObjectStore
wrapped by
CachedObjectstore
.
Objects are stored against used their hashed value. |
The KvObjectStore wraps the lower-level KvStore
|
link:../src/crux/kv.clj[role=include]
The KvStore
exposes the operations Crux needs to use a Key Value store for
indexing purposes, and for storing Objects.
Implementations of the KvStore
include:
Implementation | Description |
---|---|
| Uses RocksDB and the standard Java API that ships with RocksDB |
| Uses RocksDB and a custom built JNR bridge |
| Uses LMBD via |
| Uses LMDB via |
| An in-memory KvStore |
Crux uses a replayable transaction log that is used to derive indicies.
link:../src/crux/db.clj[role=include]
The transaction log is used to write both transaction and documents to an appendable transaction log.
Implementation | Description |
---|---|
| (default), using Kafka with separate Kafka topics for both documents and transactions |
| Uses the KvStore for the transaction log in standalone mode, backed by |
The indexer indexes documents and transactions for query purposes. This might be on the back of subscriptions to Kafka topics, or by direct calls made in a single node topological setup.
link:../src/crux/db.clj[role=include]
The single implememention is crux.tx.KvIndexer
, which makes use of
the KvStore
to persist indices.
Crux has the fundamental notion of an index.
link:../src/crux/db.clj[role=include]
The two operations are seek
and next
.
The layered index exists to faciliate the idea of navigating up and down an index, in a tree like manner.
link:../src/crux/db.clj[role=include]
For example the index attribute+value+entity+content-hash
is the following tree:
digraph G { attribute->value value->entity entity->content }
open-level
gives instructions to open and move down a level. In the
above example it could be moving the index down to point at the values
within a given attribute. That is to say that if we have an attribute
:name
, the index will iterate across all values for that attribute,
until there are no more name
values.
close-level
moves the index back-up, so in the above example, we can
iterate at the higher level of attribute.
A Virtual Index comprises together multiple child indices. This is to join indices together, returning key/value pairs on where they match.
A join condition in a query could be reflected by a Virtual Index. A Virtual Index will maintain state as to where Index is currently positioned.
UnaryJoinVirtualIndex
comprises of multiple child indices and
implements both Index
and LayeredIndex
. Calling seek-values
on
it will advance all the child indices internally until they all
contain the same key. This involves calling seek-values
on each
child index until the indices match at the same level. Calling
next-values
would move all the indices along until the next common
key that all the indices share.
| 0 | 1 | 3 | 4 | 5 | 6 | |
| 0 | 2 | 3 | 5 | |||
| 2 | 3 | 4 | 5 | 6 |
In the above example, where UnaryJoinVirtualIndex
joins three
RelationVirtualIndexs
(a,b,c). Calling seek-values
would return 3
as the first value found.
Calling next-values
would jump ahead to the value 5.
Can you improve this documentation? These fine people already did:
Jon Pither, Jeremy Taylor, Mark Woodhall & Håkan RåbergEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close