RocksDB

RocksDB is often used as the data store for XTDB’s query indices, but can also be used as a transaction log and/or document store in single node clusters.

Project Dependency

In order to use RocksDB within XTDB, you must first add RocksDB as a project dependency:

deps.edn

com.xtdb/xtdb-rocksdb {:mvn/version "{xtdb_version}"}

pom.xml

<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-rocksdb</artifactId>
    <version>{xtdb_version}</version>
</dependency>

If you’re using RocksDB and seeing out-of-memory issues, we recommend setting the environment variable MALLOC_ARENA_MAX=2 - see this issue for more details.

Using RocksDB

Replace the implementation of the desired component with xtdb.rocksdb/->kv-store

JSON

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "db-dir": "/tmp/rocksdb"
    }
  },

  "xtdb/document-store": { ... },
  "xtdb/tx-log": { ... }
}

Clojure

{:xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :db-dir (io/file "/tmp/rocksdb")}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

EDN

{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :db-dir "/tmp/rocksdb"}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

It is generally advised to use independent RocksDB instances for each component, although using a single instance for the transaction log and document store is possible. Do not share the RocksDB instance used for the index store with other components as you cannot then perform XTDB version upgrades.

Dependencies

metrics (function, default no-op): enable RocksDB metrics.

Parameters

db-dir (required, string/File/Path): path to RocksDB data directory
sync? (boolean, default false): sync to disk after every write
disable-wal? (boolean): disables the write-ahead log
db-options (RocksDB Options object): extra options to pass directly to RocksDB.

Monitoring RocksDB

To include RocksDB metrics in monitoring, override the metrics dependency:

JSON

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "metrics": {
        "xtdb/module": "xtdb.rocksdb.metrics/->metrics"
      }
      ...
    }
  },

  "xtdb/document-store": { ... },
  "xtdb/tx-log": { ... }
}

Clojure

{:xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :metrics {:xtdb/module 'xtdb.rocksdb.metrics/->metrics}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

EDN

{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :metrics {:xtdb/module xtdb.rocksdb.metrics/->metrics}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

Parameters

instance-name (string, default "rocksdb"): unique name for this instance of RocksDB, used in metrics domains
sample-window (duration, default 3s): sample window of statistics collector

Configuring the Block Cache

To configure the block cache used by the RocksDB instance, override the block-cache dependency. In the example below, there is a single shared cache between multiple kv-stores:

JSON

{
  "xtdb.rocksdb/block-cache": {
    "xtdb/module": "xtdb.rocksdb/>lru-block-cache",
    "cache-size":536870912
  },
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
      ...
    }
  },
  "xtdb/document-store": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
    }
  },
  "xtdb/tx-log": {
    "kv-store": {
      "xtdb/module": "xtdb.rocksdb/->kv-store",
      "block-cache": "xtdb.rocksdb/block-cache"
    }
  }
}

Clojure

{:xtdb.rocksdb/block-cache {:xtdb/module 'xtdb.rocksdb/->lru-block-cache
			                      :cache-size (* 512 1024 1024)}
 :xtdb/index-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                               :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/document-store {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                                  :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/tx-log {:kv-store {:xtdb/module 'xtdb.rocksdb/->kv-store
                          :block-cache :xtdb.rocksdb/block-cache}}}

EDN

{:xtdb.rocksdb/block-cache {:xtdb/module xtdb.rocksdb/->lru-block-cache
			                      :cache-size 536870912}
 :xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                               :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/document-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                                  :block-cache :xtdb.rocksdb/block-cache}}
 :xtdb/tx-log {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store
                          :block-cache :xtdb.rocksdb/block-cache}}}

Parameters

cache-size (int): Size of the cache in bytes - default size is 8Mb, although it is recommended this is set to a higher amount.

❮Query Engine AWS S3❯