Azure Blobs — com.xtdb/xtdb-core 1.24.0

Azure Blobs

You can use Azure’s Blob Storage as either:

XTDB’s 'document store'
An implementation of the XTDB query index checkpoint store.

Documents are serialized via Nippy.

In both cases, you will require both a Storage Account, and a Container to store their respective files. See the relevant Quickstart documentation from the Azure docs.

Authentication

Authentication for both the document store and checkpoint store components within the module is handled via the DefaultAzureCredential class - you will need to setup authentication using any of the methods listed within the Azure documentation to be able to make use of the operations inside the modules.

Whichever method of Authentication you use, ensure that they at least have the permissions to:

Read blobs from a container
Write blobs to a container

If using the checkpoint store, you will need the following permissions in addition to the above:

List blobs from a container
Delete blobs from a container

Project Dependency

In order to use Blob Storage within XTDB, you must first add this module as a project dependency:

deps.edn

com.xtdb/xtdb-azure-blobs {:mvn/version "{xtdb_version}"}

pom.xml

<dependency>
    <groupId>com.xtdb</groupId>
    <artifactId>xtdb-azure-blobs</artifactId>
    <version>{xtdb_version}</version>
</dependency>

Using Azure Blobs as a document store

Replace the implementation of the document store with xtdb.azure.blobs/->document-store

JSON

{
  "xtdb/document-store": {
    "xtdb/module": "xtdb.azure.blobs/->document-store",
    "storage-account": "your-storage-account",
    "container": "your-container-name",
    "prefix": "optional-containing-folder"
  },
}

Clojure

{:xtdb/document-store {:xtdb/module 'xtdb.azure.blobs/->document-store
                       :storage-account "your-storage-account"
                       :container "your-container-name",
                       :prefix "optional-containing-folder" }}

EDN

{:xtdb/document-store {:xtdb/module xtdb.azure.blobs/->document-store
                       :storage-account "your-storage-account"
                       :container "your-container-name",
                       :prefix "optional-containing-folder" }}

Parameters

storage-account (string, required) - Storage account name that the document store container lives under
container (string, required) - The container which the documents will be stored within
prefix (string, optional) - An optional prefix/folder name which will be used within document names (ie, could create a subfolder on a container called 'xtdb-document-store', and all of the document store files will be located under the folder)
cache-size (int): size of in-memory document cache (number of entries, not bytes)

Using Azure Blobs as a checkpoint store

Azure blobs can be used as a query index checkpoint store. For more advice/configuration concerning the checkpointer itself, see Checkpointing.

;; under :xtdb/index-store -> :kv-store -> :checkpointer
;; see the Checkpointing guide for other parameters
{:checkpointer {...
                :store {:xtdb/module 'xtdb.azure.blobs/->cp-store
                        :storage-account "..."
                        :container "..."
                        :prefix "..."
                 ...}}

Parameters

storage-account (string, required) - Storage account name that the checkpoint store container lives under
container (string, required) - The container which the checkpoints will be stored within
prefix (string, optional) - An optional prefix/folder name which will be used within checkpoint names (ie, could create a subfolder on a container called 'xtdb-checkpoints', and all of the checkpoint files will be located under the folder)