LMDB — com.github.zackteo/xtdb-http-server 1.21.1-dev-SNAPSHOT-3

LMDB

LMDB can be used as the data store for XTDB’s query indices, but can also be used as a transaction log and/or document store in single node clusters.

LMDB tends to provide faster queries than RocksDB in exchange for a slower ingest rate.

Project Dependency

In order to use LMDB within XTDB, you must first add LMDB as a project dependency:

deps.edn

com.xtdb/xtdb-lmdb {:mvn/version "{xtdb_version}"}

pom.xml

<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-lmdb</artifactId>
    <version>{xtdb_version}</version>
</dependency>

If you’re using LMDB and seeing out-of-memory issues, you will likely want to configure the JVM parameter MaxDirectMemorySize (e.g. -XX:MaxDirectMemorySize=1024m), because the default value varies by JVM and could be unbounded (e.g. in the case of openjdk-11, confirmed using -XX:+PrintFinalFlags). Ensure that the sum of MaxDirectMemorySize and Xmx memory is available from the system, and be aware that consumption of direct memory may be shared with any other processes running on the system.

Using LMDB

Replace the implementation of the desired component with xtdb.lmdb/->kv-store

JSON

{
  "xtdb/index-store": {
    "kv-store": {
      "xtdb/module": "xtdb.lmdb/->kv-store",
      "db-dir": "/tmp/lmdb"
    }
  },

  "xtdb/document-store": { ... },
  "xtdb/tx-log": { ... }
}

Clojure

{:xtdb/index-store {:kv-store {:xtdb/module 'xtdb.lmdb/->kv-store
                               :db-dir (io/file "/tmp/lmdb")}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

EDN

{:xtdb/index-store {:kv-store {:xtdb/module xtdb.lmdb/->kv-store
                               :db-dir "/tmp/lmdb"}}
 :xtdb/document-store {...}
 :xtdb/tx-log {...}}

It is generally advised to use independent LMDB instances for each component, although using a single instance for the transaction log and document store is possible. Do not share the LMDB instance used for the index store with other components as you cannot then perform XTDB version upgrades.

Parameters

db-dir (required, string/File/Path): path to LMDB data directory
sync? (boolean, default false): sync to disk after every write
env-flags (int): LMDB flags
env-mapsize(int): LMDB map size