Confluent’s latest Kafka-as-a-service offering is extremely compelling for
small Crux deployments as there are no minimum fees and the pricing structure
is very simple. Monthly cost = Data in + Data out + Data retained
. There is
no need to think about brokers or other infrastructure costs. Additionally you
only pay for what you use and there are no upfront costs or termination fees.
The service is currently available in GCP and AWS.
See Confluent’s page for pricing details: https://www.confluent.io/confluent-cloud/
Follow the short sequence of sign-up steps to create an account: https://www.confluent.io/confluent-cloud/
You will need to provide a valid credit/debit card in order to create an account.
Login to access your environment dashboard.
You will need to choose a name (e.g. crux-1
) cloud provider (e.g. GCP
) and
region (e.g. London
).
Under .properties
file
in a safe location that is accessible from your Crux REPL.
crux-kafka
must be provided. For instance, navigate to crux-dev
and run
lein repl
Update the various values based on your configuration and run the following code to connect to your cluster and make a transaction:
(require '[crux.api :as crux])
(import (crux.api ICruxAPI))
(def ^crux.api.ICruxAPI system
(crux/start-cluster-node
{:kv-backend "crux.kv.memdb.MemKv"
:tx-topic "tx-1" ; choose your tx-topic name
:doc-topic "doc-1" ; choose your doc-topic name
:replication-factor 3 ; Confluent Cloud requires this to be `3`
:doc-partitions 6 ; Confluent Cloud default -- you are limited to 2048 partitions per topic for a non-Enterprise cluster
:bootstrap-servers "" ; your `bootstrap.servers` value found in the generated properties file
:kafka-properties-file "path/to/my-kafka.properties"})) ; the path of the your properties file
; you can also use `:kafka-properties-map` although this increases the risk of
; checking-in your SASL secrets!
(def template-kafka-properties-map
{"ssl.endpoint.identification.algorithm" "https"
"sasl.mechanism" "PLAIN"
"request.timeout.ms" "20000"
"bootstrap.servers" "" ; set to your "url:port" without specifying a protocol
"retry.backoff.ms" "500"
"sasl.jaas.config" "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
; note the escaped string characters surrounding your username and password SASL values
"security.protocol" "SASL_SSL"})
(crux/submit-tx system [[:crux.tx/put {:crux.db/id :my-first-doc :very "cool"}]])
(crux/q (crux/db system) {:find '[e] :where '[[e :crux.db/id _]] :full-results? true})
Note that Crux will automatically generate topics with the required
retention/compaction configurations and set the partition number for the
transaction topic to 1
.
You can also create and manage topics independently of Crux using a CLI tool or the Confluent Cloud web interface, but they will need to be configured appropriately (see: https://github.com/juxt/crux/blob/master/crux-kafka/src/crux/kafka.clj).
Can you improve this documentation?Edit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close