Google Cloud Storage — com.github.zackteo/xtdb-http-server 1.21.1-dev-SNAPSHOT-3

Google Cloud Storage

You can use Google’s Cloud Storage (GCS) as XTDB’s 'document store' or 'checkpoint store'.

Documents are serialized via Nippy.

Project Dependency

In order to use GCS within XTDB, you must first add this module as a project dependency:

deps.edn

com.xtdb/xtdb-google-cloud-storage {:mvn/version "{xtdb_version}"}

pom.xml

<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-google-cloud-storage</artifactId>
    <version>{xtdb_version}</version>
</dependency>

Using GCS

Replace the implementation of the document store with xtdb.google.cloud-storage/->document-store

JSON

{
  "xtdb/document-store": {
    "xtdb/module": "xtdb.google.cloud-storage/->document-store",
    "root-path": "gs://bucket/prefix"
  },
}

Clojure

{:xtdb/document-store {:xtdb/module 'xtdb.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}

EDN

{:xtdb/document-store {:xtdb/module xtdb.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}

Follow the GCS Authentication Guide to get set up.

Parameters

root-path (string/Path, required): path where documents will be stored, gs://bucket/prefix
cache-size (int): size of in-memory document cache (number of entries, not bytes)
pool-size (int, default 4): size of thread-pool for GCS operations

Checkpoint store

GCS can be used as a query index checkpoint store.

Checkpoints aren’t GC’d by XTDB - we recommend you set a lifecycle policy on GCS to remove older checkpoints.

;; under :xtdb/index-store -> :kv-store -> :checkpointer
;; see the Checkpointing guide for other parameters
{:checkpointer {...
                :store {:xtdb/module 'xtdb.google.cloud-storage/->checkpoint-store
                        :path "gs://bucket/prefix"}}

Parameters

path (string/URI, required): URI of the form "gs://bucket/prefix"