Confluent’s latest Kafka-as-a-service offering is extremely compelling for
small Crux deployments as there are no minimum fees and the pricing structure
is very simple. Monthly cost = Data in + Data out + Data retained
. There is
no need to think about brokers or other infrastructure costs. Additionally you
only pay for what you use and there are no upfront costs or termination fees.
The service is currently available in GCP and AWS.
See Confluent’s page for pricing details: https://www.confluent.io/confluent-cloud/
Follow the short sequence of sign-up steps to create an account: https://www.confluent.io/confluent-cloud/
You will need to provide a valid credit/debit card in order to create an account.
Login to access your environment dashboard.
You will need to choose a name (e.g. crux-1
) cloud provider (e.g. GCP
) and
region (e.g. London
).
Under .properties
file
in a safe location that is accessible from your Crux REPL.
crux-kafka
must be provided. Run lein repl
Update the various values based on your configuration and run the following code to connect to your cluster and make a transaction:
(require '[crux.api :as crux])
(import (crux.api ICruxAPI))
(def ^crux.api.ICruxAPI node
(crux/start-node
{:crux.kafka/kafka-config {:bootstrap-servers "" ; the `bootstrap.servers` value found in your generated properties file
:kafka-properties-file "path/to/my-kafka.properties"}
:crux/tx-log {:crux/module 'crux.kafka/->tx-log
:kafka-config :crux.kafka/kafka-config
:tx-topic-opts {:replication-factor 3 ; Confluent Cloud requires this to be `3`
:topic-name "tx-1"}}
:crux/document-store {:crux/module 'crux.kafka/->document-store
:kafka-config :crux.kafka/kafka-config
:doc-topic-opts {:replication-factor 3
:topic-name "doc-1" ; choose your doc-topic name
:doc-partitions 6} ; Confluent Cloud default
:local-document-store {:kv-store :rocksdb}}
:crux/index-store {:kv-store :rocksdb}
:rocksdb {:crux/module 'crux.rocksdb/->kv-store
:db-dir "/tmp/rocksdb"}}))
; you can also use `:kafka-properties-map` although this increases the risk of
; checking-in your SASL secrets!
(def template-kafka-properties-map
{"ssl.endpoint.identification.algorithm" "https"
"sasl.mechanism" "PLAIN"
"request.timeout.ms" "20000"
"bootstrap.servers" "" ; set to your "url:port" without specifying a protocol
"retry.backoff.ms" "500"
"sasl.jaas.config" "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
; note the escaped string characters surrounding your username and password SASL values
"security.protocol" "SASL_SSL"})
(crux/submit-tx node [[:crux.tx/put {:crux.db/id :my-first-doc :very "cool"}]])
(crux/q (crux/db node) {:find '[e] :where '[[e :crux.db/id _]] :full-results? true})
Note that Crux will automatically generate topics with the required
retention/compaction configurations and set the partition number for the
transaction topic to 1
.
You can also create and manage topics independently of Crux using a CLI tool or the Confluent Cloud web interface, but they will need to be configured appropriately (see: https://github.com/juxt/crux/blob/master/crux-kafka/src/crux/kafka.clj).
Can you improve this documentation? These fine people already did:
James Henderson, Jeremy Taylor, Daniel Mason & Tom TaylorEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close