Configuration — juxt/crux-core 20.09-1.12.0-beta-SNAPSHOT

Configuration

A Crux node consists of a number of modules, which can be independently configured and augmented.

Once you have an in-memory Crux node set up, you can then start to configure the various modules - either through a JSON config file, EDN config file, or programatically:

On the command line, you can supply a JSON/EDN configuration file using -f <file>

For a Java in-process node, the modules are configured using the supplied Configurator, a file, or a classpath resource:

import crux.api.Crux;

ICruxAPI cruxNode = Crux.startNode(new File(...));

ICruxAPI cruxNode = Crux.startNode(MyApp.class.getResource(...));

ICruxAPI cruxNode = Crux.startNode(n -> {
   // ...
});

For a Clojure in-process node, the start-node function accepts a module tree, a file, or a resource.

(require '[crux.api :as crux]
         '[clojure.java.io :as io])

(crux/start-node {...}) ; module tree
(crux/start-node (io/file ...))
(crux/start-node (io/resource ...))

Without any explicit configuration, Crux will start an in-memory node.

At this point, you can start submitting transactions and running queries!

Modules

Crux has three main pluggable components - the transaction log, the document store, and the query index store. All three are backed by local KV stores by default, but they can be independently configured and overridden - you might choose to host the transaction log in Kafka, the document store in AWS’s S3, and the query indices in RocksDB.

	Transaction Log	Document Store	Index Store
Kafka	✓	✓
JDBC	✓	✓
In-memory KV	✓	✓	✓
LMDB (KV)	✓	✓	✓
RocksDB (KV)	✓	✓	✓
S3		✓

Transaction Log

Document Store

Index Store

Kafka

✓

JDBC

✓

In-memory KV

✓

LMDB (KV)

✓

RocksDB (KV)

✓

For specific details and examples of how to configure each of these modules, see their individual sections.

Each module has both an underlying implementation and overridable parameters - for each module, you can choose to keep the implementation and override its parameters, or you can choose to override the implementation entirely.

To add the HTTP server module, and specify its port:

Java

ICruxAPI cruxNode = Crux.startNode(n -> {
   n.with("crux.http-server/server", http -> {
     http.set("port", 3000);
   });
});

JSON

{
  "crux.http-server/server": {
    "port": 3000
  }
}

Clojure/EDN

{:crux.http-server/server {:port 3000}}

Overriding the module implementation

To override the underlying implementation, specify the factory function of the new implementation. For example, using S3’s crux.s3/->document-store factory:

Java

ICruxAPI cruxNode = Crux.startNode(n -> {
   n.with("crux/document-store", docStore -> {
     docStore.module("crux.s3/->document-store");
     docStore.set("bucket", "my-bucket");
     docStore.set("prefix", "my-prefix");
   });
});

JSON

{
  "crux/document-store": {
    "crux/module": "crux.s3/->document-store",
    "bucket": "my-bucket",
    "prefix": "my-prefix"
  }
}

Clojure/EDN

{:crux/document-store {:crux/module 'crux.s3/->document-store
                       :bucket "my-bucket"
                       :prefix "my-prefix"}}

Nested modules

Modules in Crux form an arbitrarily nested tree - parent modules depend on child modules. For example, the default implementations of the three main Crux modules are KV store backed implementations - the KV transaction log, the KV document store and the KV index store. Each of these implementations depends on being given a concrete KV store implementation - by default, an in-memory KV store. To override the implementation and parameters of this KV store (for example, to replace it with RocksDB), we override its kv-store dependency, replacing the implementation of the nested module:

Java

ICruxAPI cruxNode = Crux.startNode(n -> {
  n.with("crux/tx-log", txLog -> {
     txLog.with("kv-store", kv -> {
       kv.module("crux.rocksdb/->kv-store");
       kv.set("db-dir", new File("/tmp/rocksdb"));
    });
  });
  n.with("crux/document-store", docStore -> { ... });
  n.with("crux/index-store", indexStore -> { ... });
});

JSON

{
  "crux/tx-log": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "/tmp/txs"
    }
  },

  "crux/document-store": { ... },
  "crux/index-store": { ... }
}

Clojure/EDN

{:crux/tx-log {:kv-store {:crux/module 'crux.rocksdb/->kv-store
                          :db-dir (io/file "/tmp/txs")}}
 :crux/document-store {...}
 :crux/index-store {...}}

The tx-log and document-store are considered 'golden stores'. The query indices can, should you wish to, be thrown away and rebuilt from these golden stores.

Ensure that you either persist both or neither of these golden stores. If not, Crux will work fine until you restart the node, at which point some will evaporate, but others will remain. Crux tends to get rather confused in this situation!

Likewise, if you persist the query indices, you’ll need to persist both the golden stores.

Sharing modules - references

When two modules depend on a similar type of module, by default, they get an instance each. For example, if we were to write the following, the transaction log and the document store would get their own RocksDB instance:

{
  "crux/tx-log": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "/tmp/txs"
    }
  },
  "crux/document-store": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "/tmp/docs"
    }
  }
}

We can store both the transaction log and the document store in the same KV store, to save ourselves some hassle. We specify a new top-level module, and then refer to it by name where required:

Java

ICruxAPI cruxNode = Crux.startNode(n -> {
   n.with("my-rocksdb", rocks -> {
     rocks.module("crux.rocksdb/->kv-store");
     rocks.set("db-dir", new File("/tmp/rocksdb"));
   });
   n.with("crux/document-store", docStore -> {
     docStore.with("kv-store", "my-rocksdb");
   });
   n.with("crux/tx-log", txLog -> {
     txLog.with("kv-store", "my-rocksdb");
   });
});

JSON

{
  "my-rocksdb": {
    "crux/module": "crux.rocksdb/->kv-store",
    "db-dir": "/tmp/txs"
  },
  "crux/tx-log": {
    "kv-store": "my-rocksdb"
  },
  "crux/document-store": {
    "kv-store": "my-rocksdb"
  }
}

Clojure/EDN

{:my-rocksdb {:crux/module 'crux.rocksdb/->kv-store
              :db-dir (io/file "/tmp/rocksdb")}
 :crux/tx-log {:kv-store :my-rocksdb}
 :crux/document-store {:kv-store :my-rocksdb}}

Writing your own module (Clojure)

Crux modules are (currently) vanilla 1-arg Clojure functions with some optional metadata to specify dependencies and arguments. By convention, these are named ->your-component, to signify that it’s returning an instance of your component. If the value returned implements AutoCloseable/Closeable, the module will be closed when the Crux node is stopped.

The most basic component would be just a Clojure function, returning the started module:

(defn ->server [opts]
  ;; start your server
  )

You can specify arguments using the :crux.system/args metadata key - this example declares a required :port option, checked against the given spec, defaulting to 3000:

(require '[crux.system :as sys])

(defn ->server {::sys/args {:port {:spec ::sys/int
                                   :doc "Port to start the server on"
                                   :required? true
                                   :default 3000}}}
  [{:keys [port] :as options}]

  ;; start your server
  )

You can specify dependencies using :crux.system/deps - a map of the dependency key to its options. The options takes the same form as the end-user options - you can specify :crux/module for the default implementation, as well as any parameters. The started dependencies are passed to you as part of the function’s parameter, with the args. Bear in mind that any options you do specify can be overridden by end-users!

(defn ->server {::sys/deps {:other-module {:crux/module `->other-module
                                           :param "value"}
                            ...}}
  [{:keys [other-module]}]
  ;; start your server
  )

You can also use refs - for example, to depend on the Crux node:

(defn ->server {::sys/deps {:crux-node :crux/node}
                ::sys/args {:spec ::sys/int
                            :doc "Port to start the server on"
                            :required? true
                            :default 3000}}
  [{:keys [crux-node] :as options}]
  ;; start your server
  )