jackdaw.client.partitioning
Extras for jackdaw.client which help you define record partitioning
schemes as part of a topic's configuration.
The partitioning API provided by Kafka, like the serdes API, leaves
a lot to be desired when trying to interop from Clojure. You have to
define a org.apache.kafka.clients.producer.Partitioner class
implementation with an 0-arity constructor, and you include the name
of that Partitioner class in your producer options. This seems to
have been done so that the Partitioner can have access to Kafka
internal state about the cluster, from which to read partition count
and related data. But this pretty soundly defeats Clojure's idioms
of avoiding class generation wherever possible and using instance
parameterization.
The Producer (and Consumer) APIs however do expose
.partitionsFor - a way to interrogate a topic to understand how
many partitions it contains.
This namespace defines a mechanism by which clients can define "defaulting" behavior both for record keys, and for record partitioning.
Lets say I want to specify that my topic is always partitioned by some field of the records on the topic. It would be convenient to let a (thin) framework handle that.
Likewise it would be convenient to easily define as normal Clojure functions the computation by which I wish to assign records to partitions, rather than having to code up a custom class.
This namespace provides both capabilities via an extended
#'->ProducerRecord, and provides a #'produce! identical to that
in jackdaw.client but backed by the partitioning machinery.
Extras for `jackdaw.client` which help you define record partitioning schemes as part of a topic's configuration. The partitioning API provided by Kafka, like the serdes API, leaves a lot to be desired when trying to interop from Clojure. You have to define a `org.apache.kafka.clients.producer.Partitioner` class implementation with an 0-arity constructor, and you include the name of that Partitioner class in your producer options. This seems to have been done so that the Partitioner can have access to Kafka internal state about the cluster, from which to read partition count and related data. But this pretty soundly defeats Clojure's idioms of avoiding class generation wherever possible and using instance parameterization. The `Producer` (and `Consumer`) APIs however do expose `.partitionsFor` - a way to interrogate a topic to understand how many partitions it contains. This namespace defines a mechanism by which clients can define "defaulting" behavior both for record keys, and for record partitioning. Lets say I want to specify that my topic is always partitioned by some field of the records on the topic. It would be convenient to let a (thin) framework handle that. Likewise it would be convenient to easily define as normal Clojure functions the computation by which I wish to assign records to partitions, rather than having to code up a custom class. This namespace provides both capabilities via an extended `#'->ProducerRecord`, and provides a `#'produce!` identical to that in `jackdaw.client` but backed by the partitioning machinery.
default-partitionclj
(default-partition {:keys [topic-name key-serde]} key value partitions)The kafka default partitioner. As a ::partition-fn
The kafka default partitioner. As a `::partition-fn`
default-partitioner*clj
(default-partitioner* key-bytes num-partitions)Mimics the kafka default partitioner.
Mimics the kafka default partitioner.
produce!clj
(produce! producer topic value)(produce! producer topic key value)(produce! producer topic partition key value)(produce! producer topic partition timestamp key value)(produce! producer topic partition timestamp key value headers)Like #'jackdaw.client/produce! but used the partitioning machinery
if possible rather than just building a ProducerRecord.
Returns a future which will produce datafied record metadata when forced.
Like `#'jackdaw.client/produce!` but used the partitioning machinery if possible rather than just building a `ProducerRecord`. Returns a future which will produce datafied record metadata when forced.
record-key->key-fnclj
(record-key->key-fn {:keys [record-key] :as t})Given a topic config having :record-key, parse it,
annotating the topic with a :Lkey-fn which will simply fetch the
specified record-key out of any record.
Given a topic config having `:record-key`, parse it, annotating the topic with a `:Lkey-fn` which will simply fetch the specified record-key out of any record.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close