Development Setup

Kafka Metamorphosis Development Guide

Prerequisites

1. Java 8 or higher
2. Leiningen (Clojure build tool)
3. Apache Kafka (for testing)

Installation

Windows with WSL

If you're using WSL (Windows Subsystem for Linux), install the prerequisites in your WSL environment:

# Update package list
sudo apt update

# Install Java
sudo apt install openjdk-11-jdk

# Install Leiningen
curl -O https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
chmod +x lein
sudo mv lein /usr/local/bin/
lein --version

# Install Kafka (optional, for testing) — Kafka 4.3.0 (KRaft mode, no ZooKeeper)
wget https://downloads.apache.org/kafka/4.3.0/kafka_2.13-4.3.0.tgz
tar -xzf kafka_2.13-4.3.0.tgz

macOS

# Install Java
brew install openjdk@11

# Install Leiningen
brew install leiningen

# Install Kafka (optional)
brew install kafka

Linux

# Install Java
sudo apt install openjdk-11-jdk  # Ubuntu/Debian
# or
sudo yum install java-11-openjdk  # CentOS/RHEL

# Install Leiningen
curl -O https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
chmod +x lein
sudo mv lein /usr/local/bin/

# Install Kafka
# Download from https://kafka.apache.org/downloads

Development Workflow

1. Clone and setup

   cd /path/to/kafka-metamorphosis
   lein deps  # Download dependencies
   

2. Run tests

   lein test
   

3. Start REPL

   lein repl
   

4. Build JAR

   lein uberjar
   

Testing with Kafka

To test the library with a real Kafka instance. From Kafka 4.0 onwards ZooKeeper has been removed — use KRaft mode:

1. Format storage (one time only)

   KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
   bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/kraft/server.properties
   

2. Start the broker (KRaft, combined broker+controller)

   bin/kafka-server-start.sh config/kraft/server.properties
   

3. Create test topic

   bin/kafka-topics.sh --create --topic metamorphosis-topic --bootstrap-server localhost:9092
   

4. Run examples

   (require '[kafka-metamorphosis.examples :as examples])
   (examples/run-examples)
   

Single-node KRaft: required replication-factor overrides

The stock config/kraft/server.properties keeps the default replication factor of 3 for internal topics, which fails on a single-broker cluster — consumers will silently receive zero records because __consumer_offsets cannot be created. Add these overrides to config/kraft/server.properties for local dev:

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
share.coordinator.state.topic.replication.factor=1
share.coordinator.state.topic.min.isr=1

Symptom if missing: broker logs show Auto topic creation failed for __consumer_offsets with error 'INVALID_REPLICATION_FACTOR' and every consumer (including kafka-console-consumer.sh) returns 0 messages.

REPL Development

;; Start REPL and load the library
(require '[kafka-metamorphosis.producer :as producer])
(require '[kafka-metamorphosis.consumer :as consumer])

;; Create a producer (requires running Kafka)
(def p (producer/create {:bootstrap-servers "localhost:9092"
                         :key-serializer "org.apache.kafka.common.serialization.StringSerializer"
                         :value-serializer "org.apache.kafka.common.serialization.StringSerializer"}))

;; Send a test message
(producer/send! p "test-topic" "key1" "Hello from REPL!")

;; Close producer
(producer/close! p)

File Structure

src/kafka_metamorphosis/
├── core.clj           # Main namespace and exports
├── producer.clj       # Producer functions
├── consumer.clj       # Consumer functions
├── util.clj          # Utility functions
└── examples.clj      # Usage examples

test/kafka_metamorphosis/
├── core_test.clj     # Core tests
└── producer_test.clj # Producer tests

Next Steps

The basic wrapper is now implemented with:

• ✅ Producer creation and message sending (sync/async)
• ✅ Consumer creation and message polling
• ✅ Configuration utilities (map -> properties conversion)
• ✅ Kebab-case to dot notation conversion
• ✅ Basic error handling
• ✅ Resource management (close functions)

Future enhancements:

• Schema Registry support
• Kafka Streams wrapper
• Administrative operations
• Metrics integration
• Transaction support