juxt/crux-core {:mvn/version "20.09-1.11.0-beta"}
juxt/crux-kafka {:mvn/version "20.09-1.11.0-beta"}
juxt/crux-http-server {:mvn/version "20.09-1.11.0-alpha"}
juxt/crux-rocksdb {:mvn/version "20.09-1.11.0-beta"}
Use confluent-hub install juxt/kafka-connect-crux:20.09-1.11.0-beta
to download and install the connector from Confluent hub. The downloaded connector is then placed within your confluent install’s 'share/confluent-hub-components' folder.
The connector can be used as either a source or a sink. In either case, there should be an associated Crux node to communicate with.
To use our connector, you must first have a Crux node connected to Kafka. To do this, we start by adding the following dependencies to a project:
juxt/crux-core {:mvn/version "20.09-1.11.0-beta"}
juxt/crux-kafka {:mvn/version "20.09-1.11.0-beta"}
juxt/crux-http-server {:mvn/version "20.09-1.11.0-alpha"}
juxt/crux-rocksdb {:mvn/version "20.09-1.11.0-beta"}
Ensure first that you have a running Kafka broker to connect to. We import the dependencies into a file or REPL, then create our Kafka connected 'node' with an associated http server for the connector to communicate with:
(require '[crux.api :as crux]
'[crux.http-server :as srv])
(import (crux.api ICruxAPI))
(def ^crux.api.ICruxAPI node
(crux/start-node {:crux.node/topology '[crux.kafka/topology crux.http-server/module]
:crux.kafka/bootstrap-servers "localhost:9092"
:crux.http-server/port 3000}))
Run the following command within the base of the Confluent folder, to create a worker which connects to the 'connect-test' topic, ready to send messages to the node. This also makes use of connect-file-source, checking for changes in a file called 'test.txt':
./bin/connect-standalone etc/kafka/connect-standalone.properties share/confluent-hub-components/juxt-kafka-connect-crux/etc/local-crux-sink.properties etc/kafka/connect-file-source.properties
Run the following within your Confluent directory, to add a line of JSON to 'test.txt':
echo '{"crux.db/id": "415c45c9-7cbe-4660-801b-dab9edc60c84", "value": "baz"}' >> test.txt
Now, verify that this was transacted within your REPL:
(crux/entity (crux/db node) "415c45c9-7cbe-4660-801b-dab9edc60c84")
==>
{:crux.db/id #uuid "415c45c9-7cbe-4660-801b-dab9edc60c84", :value "baz"}
Run the following command within the base of the Confluent folder, to create a worker connects to the 'connect-test' topic, ready to receive messages from the node. This also makes use of 'connect-file-sink', outputting transactions to your node within 'test.sink.txt':
./bin/connect-standalone etc/kafka/connect-standalone.properties share/confluent-hub-components/juxt-kafka-connect-crux/etc/local-crux-source.properties etc/kafka/connect-file-sink.properties
Within your REPL, transact an element into Crux:
(crux/submit-tx node [[:crux.tx/put {:crux.db/id #uuid "415c45c9-7cbe-4660-801b-dab9edc60c82", :value "baz-source"}]])
Check the contents of 'test.sink.txt' using the command below, and you should see that the transactions were outputted to the 'connect-test' topic:
tail test.sink.txt ==> [[:crux.tx/put {:crux.db/id #uuid "415c45c9-7cbe-4660-801b-dab9edc60c82", :value "baz-source"} #inst "2019-09-19T12:31:21.342-00:00"]]
Can you improve this documentation?Edit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close