Configuration

A XTDB node consists of a number of modules, which can be independently configured and augmented.

Once you have an in-memory XTDB node set up, you can then start to configure the various modules - either through a JSON config file, EDN config file, or programmatically:

Command Line

On the command line, you can supply a JSON/EDN configuration file using -f <file>

Java

For a Java in-process node, the modules are configured using the supplied Configurator, a file, or a classpath resource:

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

Kotlin

For a Kotlin in-process node, the modules are configured using the supplied Configurator, a file, or a classpath resource:

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

Clojure

For a Clojure in-process node, the start-node function accepts a module tree, a file, or a resource.

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

Without any explicit configuration, XTDB will start an in-memory node.

At this point, you can start submitting transactions and running queries!

Modules

XTDB has three main pluggable components - the transaction log, the document store, and the query index store. All three are backed by local KV stores by default, but they can be independently configured and overridden - you might choose to host the transaction log in Kafka, the document store in AWS’s S3, and the query indices in RocksDB.

	Transaction Log	Document Store	Index Store
AWS S3		✓
Azure Blobs		✓
DLT - Corda ^[1]	✓	✓
Google Cloud Storage		✓
Kafka	✓	✓
JDBC	✓	✓
In-memory KV	✓	✓	✓
LMDB (KV)	✓	✓	✓
RocksDB (KV)	✓	✓	✓
Xodus (KV) ^[2]	✓	✓	✓

Transaction Log

Document Store

Index Store

AWS S3

✓

Azure Blobs

✓

DLT - Corda ^[1]

✓

Google Cloud Storage

✓

Kafka

✓

JDBC

✓

In-memory KV

✓

LMDB (KV)

✓

RocksDB (KV)

✓

Xodus (KV) ^[2]

✓

For specific details and examples of how to configure each of these modules, see their individual sections.

Each module has both an underlying implementation and overridable parameters - for each module, you can choose to keep the implementation and override its parameters, or you can choose to override the implementation entirely.

To add the HTTP server module, and specify its port:

Java

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

Kotlin

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

JSON

link:example$test/xtdb/docs/examples/configuration/config.json[role=include]

Clojure

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

EDN

link:example$test/xtdb/docs/examples/configuration/config.edn[role=include]

Overriding the module implementation

To override the underlying implementation, specify the factory function of the new implementation. For example, using S3’s xtdb.s3/->document-store factory:

Java

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

Kotlin

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

JSON

link:example$test/xtdb/docs/examples/configuration/config_override.json[role=include]

Clojure

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

EDN

link:example$test/xtdb/docs/examples/configuration/config_override.edn[role=include]

Nested modules

Modules in XTDB form an arbitrarily nested tree - parent modules depend on child modules. For example, the default implementations of the three main XTDB modules are KV store backed implementations - the KV transaction log, the KV document store and the KV index store. Each of these implementations depends on being given a concrete KV store implementation - by default, an in-memory KV store. To override the implementation and parameters of this KV store (for example, to replace it with RocksDB), we override its kv-store dependency, replacing the implementation of the nested module:

Java

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]
link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]
link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

Kotlin

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]
link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]
link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

JSON

link:example$test/xtdb/docs/examples/configuration/config_nested.json[role=include]

Clojure

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]
link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]
link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

EDN

link:example$test/xtdb/docs/examples/configuration/config_nested.edn[role=include]

The tx-log and document-store are considered 'golden stores'. The query indices can, should you wish to, be thrown away and rebuilt from these golden stores.

Ensure that you either persist both or neither of these golden stores. If not, XTDB will work fine until you restart the node, at which point some will evaporate, but others will remain. XTDB tends to get rather confused in this situation!

Likewise, if you persist the query indices, you’ll need to persist both the golden stores.

Sharing modules - references

When two modules depend on a similar type of module, by default, they get an instance each. For example, if we were to write the following, the transaction log and the document store would get their own RocksDB instance:

link:example$test/xtdb/docs/examples/configuration/sharing_modules_naive.json[role=include]

We can store both the transaction log and the document store in the same KV store, to save ourselves some hassle. We specify a new top-level module, and then refer to it by name where required:

Java

link:example$test/xtdb/docs/examples/configuration/ConfigurationTest.java[role=include]

Kotlin

link:example$test/xtdb/docs/examples/configuration/KConfigurationTest.kt[role=include]

JSON

link:example$test/xtdb/docs/examples/configuration/config_shared.json[role=include]

Clojure

link:example$test/xtdb/docs/examples/configuration/configuration_test.clj[role=include]

EDN

link:example$test/xtdb/docs/examples/configuration/config_shared.edn[role=include]

Writing your own module (Clojure)

XTDB modules are (currently) vanilla 1-arg Clojure functions with some optional metadata to specify dependencies and arguments. By convention, these are named ->your-component, to signify that it’s returning an instance of your component. If the value returned implements AutoCloseable/Closeable, the module will be closed when the XTDB node is stopped.

The most basic component would be just a Clojure function, returning the started module:

(defn ->server [opts]
  ;; start your server
  )

You can specify arguments using the :xtdb.system/args metadata key - this example declares a required :port option, checked against the given spec, defaulting to 3000:

(require '[xtdb.system :as sys])

(defn ->server {::sys/args {:port {:spec ::sys/int
                                   :doc "Port to start the server on"
                                   :required? true
                                   :default 3000}}}
  [{:keys [port] :as options}]

  ;; start your server
  )

You can specify dependencies using :xtdb.system/deps - a map of the dependency key to its options. The options takes the same form as the end-user options - you can specify :xtdb/module for the default implementation, as well as any parameters. The started dependencies are passed to you as part of the function’s parameter, with the args. Bear in mind that any options you do specify can be overridden by end-users!

(defn ->server {::sys/deps {:other-module {:xtdb/module `->other-module
                                           :param "value"}
                            ...}}
  [{:keys [other-module]}]
  ;; start your server
  )

You can also use refs - for example, to depend on the XTDB node:

(defn ->server {::sys/deps {:xtdb-node :xtdb/node}
                ::sys/args {:spec ::sys/int
                            :doc "Port to start the server on"
                            :required? true
                            :default 3000}}
  [{:keys [xtdb-node] :as options}]
  ;; start your server
  )

1. configured via its own entry point - see the module docs for more information

2. via third-party crux-xodus module

❮(ns xtdb.api)Corda❯