Crux-in-a-Box (on RocksDB)

Introduction

This guide will show you how to set up Crux with the following properties:

Disk: Records persist to disk
One node: Only one Crux node is configured
Storage: RocksDB is used for all 3 stores (tx-log, doc-store, and index-store)

This guide will not discuss high-availability, failover/clustering/replication, or backups. You can use this simple RocksDB configuration in production and back it up as you would any other volume. If you want more advanced features, however, you should consider the Kafka and/or JDBC modules.

All the code from this guide is compiled into minimal (working) project examples for Clojure and Java you can see in the crux-in-a-box GitHub repo.

Steps

1. Install Dependencies

Ensure you are using Java 8 or higher:

java -version

Add Crux to your Clojure or Java project:

deps.edn

juxt/crux-core {:mvn/version "{crux_version}-beta"}
juxt/crux-rocksdb {:mvn/version "{crux_version}-beta"}

project.clj

[juxt/crux-core "{crux_version}-beta"]
[juxt/crux-rocksdb "{crux_version}-beta"]

pom.xml

<dependency>
    <groupId>juxt</groupId>
    <artifactId>crux-core</artifactId>
    <version>{crux_version}-beta</version>
</dependency>
<dependency>
    <groupId>juxt</groupId>
    <artifactId>crux-rocksdb</artifactId>
    <version>{crux_version}-beta</version>
</dependency>

A Note on Logging

Ignore this note if you just want to get up and running quickly. It’s completely optional, but nice to have.

Without setting up logging, you will see a SLF4J: Defaulting to no-operation (NOP) logger implementation message on STDOUT from Crux. This is harmless, but in a real application you will want to configure Logback with the SLF4J API to see INFO messages from Crux.

To do this in Clojure:

Add Logback, SLF4J, and tools.logging as dependencies
Set the clojure.tools.logging.factory JVM option
Create a crux-in-a-box/resources/logback.xml file

To do this in Java:

Add Logback, SLF4J, and tools.logging as dependencies
Set the clojure.tools.logging.factory JVM option
Create a crux-in-a-box/src/main/resources/logback.xml file

2. Configure Crux

This guide gets you up and running quickly by hard-coding configuration values. Store these configuration values with your favourite configuration library. In Clojure, you might use Aero.

This guide also does not make any assumptions about how you manage stateful services. Crux is a stateful service, however, and you should store the Crux node in your favourite state management library. In Clojure, you might use Mount.

Clojure

(ns crux-in-a-box.db
  (:require [clojure.java.io :as io]
            [crux.api :as crux]))

(defn start-crux! []
  (letfn [(kv-store [dir]
            {:kv-store {:crux/module 'crux.rocksdb/->kv-store
                        :db-dir      (io/file dir)
                        :sync?       true}})]
    (crux/start-node
     {:crux/tx-log              (kv-store "data/dev/tx-log")
      :crux/document-store      (kv-store "data/dev/doc-store")
      :crux/index-store         (kv-store "data/dev/index-store")})))

(def crux-node (start-crux!))

(defn stop-crux! []
  (.close crux-node))

Java

package cruxinabox;

import java.io.File;
import java.io.IOException;
import crux.api.Crux;
import crux.api.ICruxAPI;

class Box {
    public static void main(String[] args) {
        try(ICruxAPI cruxNode = Crux.startNode(new File("config.json"))) {
            System.out.println("Crux Started.");
        }
        catch (IOException e) {
            // ...
        }
    }
}

config.json

{
  "crux/index-store": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "data/index-store"
    }
  },
  "crux/document-store": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "data/doc-store"
    }
  },
  "crux/tx-log": {
    "kv-store": {
      "crux/module": "crux.rocksdb/->kv-store",
      "db-dir": "data/tx-log"
    }
  },
  "crux.lucene/lucene-store": {
    "db-dir": "data/dev/lucene-dir"
  },
  "crux.http-server/server": {
    "port": 9999
  }
}

3. Take it for a test drive

Time to play with your new Crux node. Try inserting some data, query it back out, and then shut down your node. This guide is not meant to teach you Datalog, the Crux API, or Crux query semantics — you can look through the Tutorial and Reference Docs to dig deeper. But you should note that Crux is asynchronous by default. That means submit-tx returns immediately, even if your document hasn’t been indexed yet. It’s unlikely your query will come back empty, but don’t be shocked if it does. Just try it again.

Clojure REPL

crux-in-a-box.db> (crux/submit-tx crux-node [[:crux.tx/put
                                              {:crux.db/id "hi2u"
                                               :user/name "zig"}]])
;; => #:crux.tx{:tx-id 0, :tx-time #inst "2021-03-11T02:27:09.176-00:00"}
crux-in-a-box.db> (crux/q (crux/db crux-node) '{:find [e]
                                                :where [[e :user/name "zig"]]} )
;; => #{["hi2u"]}
crux-in-a-box.db> (stop-crux!)
;; => nil

Java

/* include these imports in addition to those you used to configure crux: */
import java.util.HashMap;
import java.util.List;

import crux.api.ICruxDatasource;
import crux.api.ICursor;
import crux.api.CruxDocument;
import crux.api.TransactionInstant;
import crux.api.tx.Transaction;

/* submitTx example: */
HashMap<String, Object> data = new HashMap<>();
data.put("user/name", "zig");
CruxDocument document = CruxDocument.create("hi2u", data);
TransactionInstant transaction = node.submitTx(Transaction.buildTx(tx -> {
    tx.put(document);
}));
System.out.println(data.toString());

/* query example: */
node.awaitTx(transaction, null);
String query = "{:find [e] :where [[e :user/name \"zig\"]]}";
ICruxDatasource db = node.db();
ICursor<List<?>> results = db.openQuery(query);
if (results.hasNext()) {
    List<?> result = results.next();
    System.out.println(result.toString());
}
db.close();
node.close();

Considerations

This is a simple Crux setup with a small footprint. You could use this setup to build a small greenfield project or replace an existing Postgres, MySQL, or MongoDB installation which currently supports CRUD or reference data. With Crux, that database can grow with you. When it eventually requires streaming, replication, or high availability, you can migrate from RocksDB to Kafka or JDBC. But you don’t need to involve those modules up-front unless you’re sure you need them.