Kafka with Docker - Development Guide

Kafka 4.x removes ZooKeeper. All Docker setups below are KRaft-based and use apache/kafka:4.3.0.

🐳 Quick Setup with Docker

Kafka Metamorphosis includes utility functions to facilitate local development using Docker with two KRaft-based architectures:

🏗️ Choose Your Architecture

🆕 KRaft Mode (Recommended) - Kafka + UI

(require '[kafka-metamorphosis.dev :as dev])

;; Complete setup with KRaft + Kafka UI
(dev/kafka-setup-kraft!)

;; Or just start Kafka
(dev/kafka-up-kraft!)

⚡ Simple KRaft (Fastest) - Single container

;; Minimalist setup (no UI, super fast)
(dev/kafka-setup-simple!)

;; Or just start Kafka
(dev/kafka-up-simple!)

🚀 Generic Setup

;; Generic method - choose the mode
(dev/kafka-dev-setup!)                    ; Default: :kraft
(dev/kafka-dev-setup! :kraft)             ; KRaft + UI
(dev/kafka-dev-setup! :simple)            ; Simple KRaft (no UI)
(dev/kafka-dev-setup! :kraft ["my-topic"]) ; KRaft with custom topics

📊 Mode Comparison

Mode	Containers	Startup	UI	Image	Usage
KRaft	2 (Kafka + UI)	Medium	✅	apache/kafka:4.3.0	Complete development
Simple	1 (Kafka)	Fast	❌	apache/kafka:4.3.0	Quick tests

ZooKeeper-based mode was removed because Kafka 4.x no longer supports it. Stick to KRaft.

🔧 Manual Control

;; Start Kafka
(dev/kafka-docker-up!)

;; Wait for it to be ready
(dev/wait-for-kafka)

;; Check status
(dev/kafka-docker-status)

;; View logs
(dev/kafka-docker-logs!)
(dev/kafka-docker-logs! "kafka" true)  ; Follow Kafka logs

;; Restart services
(dev/kafka-docker-restart!)

;; Stop services
(dev/kafka-docker-down!)

;; Stop and remove volumes (cleans data)
(dev/kafka-docker-down! true)

🧹 Complete Cleanup

;; Stop everything and keep data
(dev/kafka-dev-teardown!)

;; Stop everything and remove data
(dev/kafka-dev-teardown! true)

📊 Included Services

Kafka Broker

• Port: 9092 (main)
• Alternative port: 29092 (KRaft mode only)
• Address: localhost:9092
• Image: apache/kafka:4.3.0

Kafka UI (KRaft mode only)

• Port: 8080
• URL: http://localhost:8080
• Features:
  • View topics
  • Monitor messages
  • Manage consumers
  • View metrics

🐳 Generated Docker Compose (KRaft + UI)

The docker-compose.yml file generated by (dev/kafka-setup-kraft!) includes:

version: "3.8"

services:
  kafka:
    image: apache/kafka:4.3.0
    container_name: kafka-metamorphosis-kraft
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@localhost:29093"
      KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:29092"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092,PLAINTEXT_HOST://localhost:29092"
      KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    depends_on:
      - kafka
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local-kraft
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092

🛠️ Workflow de Desenvolvimento

1. Primeira vez / Setup inicial

;; Configure full environment
(dev/kafka-dev-setup!)

2. Daily development

;; Check if it's running
(dev/kafka-docker-status)

;; If not, start it
(dev/kafka-docker-up!)

;; Work with topics and messages...
(dev/setup-dev-topic "my-project")
(dev/send-test-messages "my-project" 10)
(dev/read-test-messages "my-project")

3. Debug / Troubleshooting

;; View Kafka logs
(dev/kafka-docker-logs! "kafka")

;; Restart if necessary
(dev/kafka-docker-restart! "kafka")

;; Check if can connect
(dev/wait-for-kafka 30)

4. End of work

;; Stop but keep data
(dev/kafka-docker-down!)

;; Or clean everything
(dev/kafka-dev-teardown! true)

🐛 Troubleshooting

Kafka won't start

# Check if Docker is running
docker --version

# Check occupied ports
netstat -an | grep ":9092"

Error logs

;; View all logs
(dev/kafka-docker-logs!)

;; View specific logs
(dev/kafka-docker-logs! "kafka")

Clean corrupted state

;; Stop everything and remove volumes
(dev/kafka-dev-teardown! true)

;; Recreate from scratch
(dev/kafka-dev-setup!)

🎯 Manual Docker Commands

If you need to run Docker commands directly:

# Start services
docker-compose up -d

# View status
docker-compose ps

# View logs
docker-compose logs kafka

# Stop services
docker-compose down

# Stop and remove volumes
docker-compose down -v

🦋 The Docker Metamorphosis is Complete!

With these functions, you can easily manage a local Kafka environment for development, transforming configuration complexity into functional simplicity! 🪲➡️🦋