Crux Google Cloud Storage

You can use Google’s Cloud Storage (GCS) as Crux’s 'document store' or 'checkpoint store'.

Documents are serialized via Nippy.

Project Dependency

In order to use GCS within Crux, you must first add this module as a project dependency:

deps.edn

juxt/crux-google-cloud-storage {:mvn/version "{crux_version}-alpha"}

pom.xml

<dependency>
    <groupId>juxt</groupId>
    <artifactId>crux-google-cloud-storage</artifactId>
    <version>{crux_version}-alpha</version>
</dependency>

Using GCS

Replace the implementation of the document store with crux.google.cloud-storage/->document-store

JSON

{
  "crux/document-store": {
    "crux/module": "crux.google.cloud-storage/->document-store",
    "root-path": "gs://bucket/prefix"
  },
}

Clojure

{:crux/document-store {:crux/module 'crux.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}

EDN

{:crux/document-store {:crux/module crux.google.cloud-storage/->document-store
                       :root-path "gs://bucket/prefix"}}

Follow the GCS Authentication Guide to get set up.

Parameters

root-path (string/Path, required): path where documents will be stored, gs://bucket/prefix
cache-size (int): size of in-memory document cache
pool-size (int, default 4): size of thread-pool for GCS operations

Checkpoint store

GCS can be used as a query index checkpoint store.

Checkpoints aren’t GC’d by Crux - we recommend you set a lifecycle policy on GCS to remove older checkpoints.

;; under :crux/index-store -> :kv-store -> :checkpointer
;; see the Checkpointing guide for other parameters
{:checkpointer {...
                :store {:crux/module 'crux.google.cloud-storage/->checkpoint-store
                        :path "gs://bucket/prefix"}}

Parameters

path (string/URI, required): URI of the form "gs://bucket/prefix"