Use XTDB with Confluent's cloud Kafka platform in <5m — com.github.zackteo/xtdb-http-server 1.21.1-dev-SNAPSHOT-3

Use XTDB with Confluent’s cloud Kafka platform in <5m

Confluent’s latest Kafka-as-a-service offering is extremely compelling for small XTDB deployments as there are no minimum fees and the pricing structure is very simple. Monthly cost = Data in + Data out + Data retained. There is no need to think about brokers or other infrastructure costs. Additionally you only pay for what you use and there are no upfront costs or termination fees. The service is currently available in GCP and AWS.

See Confluent’s page for pricing details: https://www.confluent.io/confluent-cloud/

Setup a Confluent Cloud account

Follow the short sequence of sign-up steps to create an account: https://www.confluent.io/confluent-cloud/

You will need to provide a valid credit/debit card in order to create an account.

Create a cluster in your default environment

You will need to choose a name (e.g. xtdb-1) cloud provider (e.g. GCP) and region (e.g. London).

Create an API key

Under Data In/Out Clients click "Create Kafka Cluster API key & secret" and copy the credentials now embedded in the configuration snippet into a .properties file in a safe location that is accessible from your XTDB REPL.

Start a Clojure REPL

The XTDB kafka module must be provided. Run lein repl

Update the various values based on your configuration and run the following code to connect to your cluster and make a transaction:

(require '[xtdb.api :as xt])
(import (xtdb.api IXtdb))

(def ^xtdb.api.IXtdb node
  (xt/start-node
   {:xtdb.kafka/kafka-config {:bootstrap-servers "" ; the `bootstrap.servers` value found in your generated properties file
                              :kafka-properties-file "path/to/my-kafka.properties"}
    :xtdb/tx-log {:xtdb/module 'xtdb.kafka/->tx-log
                :kafka-config :xtdb.kafka/kafka-config
                :tx-topic-opts {:replication-factor 3 ; Confluent Cloud requires this to be `3`
                                :topic-name "tx-1"}}

    :xtdb/document-store {:xtdb/module 'xtdb.kafka/->document-store
                          :kafka-config :xtdb.kafka/kafka-config
                          :doc-topic-opts {:replication-factor 3
                                           :topic-name "doc-1" ; choose your doc-topic name
                                           :doc-partitions 6} ; Confluent Cloud default
                          :local-document-store {:kv-store :rocksdb}}
    :xtdb/index-store {:kv-store :rocksdb}
    :rocksdb {:xtdb/module 'xtdb.rocksdb/->kv-store
              :db-dir "/tmp/rocksdb"}}))

; you can also use `:kafka-properties-map` although this increases the risk of
; checking-in your SASL secrets!

(def template-kafka-properties-map
  {"ssl.endpoint.identification.algorithm" "https"
   "sasl.mechanism" "PLAIN"
   "request.timeout.ms" "20000"
   "bootstrap.servers" "" ; set to your "url:port" without specifying a protocol
   "retry.backoff.ms" "500"
   "sasl.jaas.config" "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
   ; note the escaped string characters surrounding your username and password SASL values
   "security.protocol" "SASL_SSL"})

(xt/submit-tx node [[::xt/put {:xt/id :my-first-doc :very "cool"}]])

(xt/q (xt/db node) {:find '[(pull e [*])] :where '[[e :xt/id _]]})

Note that XTDB will automatically generate topics with the required retention/compaction configurations and set the partition number for the transaction topic to 1.

You can also create and manage topics independently of XTDB using a CLI tool or the Confluent Cloud web interface, but they will need to be configured appropriately (see: https://github.com/xtdb/xtdb/blob/main/modules/kafka/src/xtdb/kafka.clj).