XTDB-in-a-Box (on RocksDB) — com.github.zackteo/xtdb-http-server 1.21.1-dev-SNAPSHOT-2

XTDB-in-a-Box (on RocksDB)

Introduction

This guide will show you how to set up XTDB with the following properties:

Disk: Records persist to disk
One node: Only one XTDB node is configured
Storage: RocksDB is used for all 3 stores (tx-log, doc-store, and index-store)

This guide will not discuss high-availability, failover/clustering/replication, or backups. You can use this simple RocksDB configuration in production and back it up as you would any other volume. If you want more advanced features, however, you should consider the Kafka and/or JDBC modules.

All the code from this guide is compiled into minimal (working) project examples for Clojure and Java you can see in the crux-in-a-box GitHub repo.

Steps

1. Install Dependencies

Ensure you are using Java 8 or higher:

java -version

Add XTDB to your Clojure or Java project:

deps.edn

com.xtdb/xtdb-core {:mvn/version "{xtdb_version}"}
com.xtdb/xtdb-rocksdb {:mvn/version "{xtdb_version}"}

project.clj

[com.xtdb/xtdb-core "{xtdb_version}"]
[com.xtdb/xtdb-rocksdb "{xtdb_version}"]

pom.xml

<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-core</artifactId>
    <version>{xtdb_version}</version>
</dependency>
<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-rocksdb</artifactId>
    <version>{xtdb_version}</version>
</dependency>

A Note on Logging

Ignore this note if you just want to get up and running quickly. It’s completely optional, but nice to have.

Without setting up logging, you will see a SLF4J: Defaulting to no-operation (NOP) logger implementation message on STDOUT from XTDB. This is harmless, but in a real application you will want to configure Logback with the SLF4J API to see INFO messages from XTDB.

To do this in Clojure:

Add Logback, SLF4J, and tools.logging as dependencies
Set the clojure.tools.logging.factory JVM option
Create a crux-in-a-box/resources/logback.xml file

To do this in Java:

Add Logback, SLF4J, and tools.logging as dependencies
Set the clojure.tools.logging.factory JVM option
Create a crux-in-a-box/src/main/resources/logback.xml file

2. Configure XTDB

This guide gets you up and running quickly by hard-coding configuration values. Store these configuration values with your favourite configuration library. In Clojure, you might use Aero.

This guide also does not make any assumptions about how you manage stateful services. XTDB is a stateful service, however, and you should store the XTDB node in your favourite state management library. In Clojure, you might use Mount.

Clojure

(ns xtdb-in-a-box.db
  (:require [clojure.java.io :as io]
            [xtdb.api :as xt]))

(defn start-xtdb! []
  (letfn [(kv-store [dir]
            {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                        :db-dir (io/file dir)
                        :sync? true}})]
    (xt/start-node
     {:xtdb/tx-log (kv-store "data/dev/tx-log")
      :xtdb/document-store (kv-store "data/dev/doc-store")
      :xtdb/index-store (kv-store "data/dev/index-store")})))

(def xtdb-node (start-xtdb!))

(defn stop-xtdb! []
  (.close xtdb-node))

Java

package xtdbinabox;

import java.io.File;
import java.io.IOException;
import xtdb.api.IXtdb;

class Box {
    public static void main(String[] args) {
        try(IXtdb xtdbNode = IXtdb.startNode(new File("config.json"))) {
            System.out.println("XTDB Started.");
        }
        catch (IOException e) {
            // ...
        }
    }
}

config.json

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "db-dir": "data/index-store"
    }
  },
  "xtdb/document-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "db-dir": "data/doc-store"
    }
  },
  "xtdb/tx-log": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "db-dir": "data/tx-log"
    }
  },
  "xtdb.lucene/lucene-store": {
    "db-dir": "data/dev/lucene-dir"
  },
  "xtdb.http-server/server": {
    "port": 9999
  }
}

3. Take it for a test drive

Time to play with your new XTDB node. Try inserting some data, query it back out, and then shut down your node. This guide is not meant to teach you Datalog, the XTDB API, or XTDB query semantics — you can look through the Tutorial and Reference Docs to dig deeper. But you should note that XTDB is asynchronous by default. That means submit-tx returns immediately, even if your document hasn’t been indexed yet. It’s unlikely your query will come back empty, but don’t be shocked if it does. Just try it again.

Clojure REPL

xtdb-in-a-box.db> (xt/submit-tx xtdb-node [[::xt/put
                                            {:xt/id "hi2u"
                                             :user/name "zig"}]])
;; => #::xt{:tx-id 0, :tx-time #inst "2021-03-11T02:27:09.176-00:00"}
xtdb-in-a-box.db> (xt/q (xt/db xtdb-node) '{:find [e]
                                            :where [[e :user/name "zig"]]} )
;; => #{["hi2u"]}
xtdb-in-a-box.db> (stop-xtdb!)
;; => nil

Java

/* include these imports in addition to those you used to configure XTDB: */
import java.util.HashMap;
import java.util.List;

import xtdb.api.IXtdbDatasource;
import xtdb.api.ICursor;
import xtdb.api.XtdbDocument;
import xtdb.api.TransactionInstant;
import xtdb.api.tx.Transaction;

/* submitTx example: */
HashMap<String, Object> data = new HashMap<>();
data.put("user/name", "zig");
XtdbDocument document = XtdbDocument.create("hi2u", data);
TransactionInstant transaction = node.submitTx(Transaction.buildTx(tx -> {
    tx.put(document);
}));
System.out.println(data.toString());

/* query example: */
node.awaitTx(transaction, null);
String query = "{:find [e] :where [[e :user/name \"zig\"]]}";
IXtdbDatasource db = node.db();
ICursor<List<?>> results = db.openQuery(query);
if (results.hasNext()) {
    List<?> result = results.next();
    System.out.println(result.toString());
}
db.close();
node.close();

Considerations

This is a simple XTDB setup with a small footprint. You could use this setup to build a small greenfield project or replace an existing Postgres, MySQL, or MongoDB installation which currently supports CRUD or reference data. With XTDB, that database can grow with you. When it eventually requires streaming, replication, or high availability, you can migrate from RocksDB to Kafka or JDBC. But you don’t need to involve those modules up-front unless you’re sure you need them.