How jeanlouisboudart/kafka-producer-benchmark works
Project Type and Purpose
• This is an *open-source benchmarking framework* that orchestrates a Kafka cluster (locally via Docker Compose or in the cloud via Terraform/Kubernetes), runs a series of producer clients (Java, .NET, Go, Python, Rust), captures metrics, and produces comparative reports.
File-Structure Highlights
• bench-initializer/ (Java): initializes topics and preloads data.
• cloud/setup/: Terraform files to provision Confluent Cloud + Kubernetes environment.
• **docker-compose*.yml**: multiple Kafka cluster topologies (Zookeeper vs KRaft, 1 vs 3 brokers).
• **\*-producer/**: one folder per client implementation, each containing a Dockerfile, language-specific code, config, and logging setup.
• jmx-exporter/, prometheus/, grafana/: monitoring stack.
• **run-scenarios*.sh, utils.sh**: orchestration scripts that discover all scenario-*.env files, build client images, launch scenarios, collect logs to target/.
Key Architectural Insights from README
• Benchmark scenarios are defined entirely by .env files.
• Clients are *pluggable*: you add a new folder with a Dockerfile and register it in utils.sh.
• Metrics flow: each producer logs to stdout → stored in target/.../*.txt; JMX-Exporter sidecars expose broker metrics → scraped by Prometheus → visualized in Grafana.
• Cloud mode uses Terraform to spin up Confluent Cloud resources and a K8s cluster, then reuses the same producer containers as Kubernetes Jobs/Pods.
Main Components & Their Interactions
a. Orchestrator
• *Local*: docker-compose.yml (and variants) + run-scenarios.sh / run-scenario.sh scripts.
• *Cloud*: Terraform (cloud/setup/*.tf) → Confluent Cloud + K8s cluster → run-scenarios-cloud.sh.
b. Kafka Cluster
• Brokers (Zookeeper-based or KRaft) running as Docker services or Cloud-managed brokers.
c. JMX-Exporter
• Attached to each broker (and optionally ZK) to expose JVM metrics on HTTP endpoints.
d. Prometheus
• Scrapes JMX-Exporter endpoints + can scrape custom metrics endpoints if producers expose any.
e. Grafana
• Dashboards provisioned under grafana/provisioning/ that query Prometheus.
f. Producer Clients
• Multiple Docker images (java-producer, dotnet-producer, golang-producer, python-producer, rust-producer-sync).
• Launched in sequence or parallel by the orchestrator scripts.
• Each logs standardized metrics to stdout (Send rate, queue time, batch size, etc.).
g. bench-initializer
• Java utility that pre-creates topics based on env variables and optionally preloads messages.
h. Reporting
• After scenarios complete, raw logs reside in target/....
• report/benchmark-report.ipynb and extract_results.py ingest these logs and produce charts.
• A Dockerized report container can build final PDF/HTML.
Architectural Patterns & Design Principles
• Containerized micro-services: each client & service in its own container.
• Plug-in architecture: new clients can be added by placing a folder + Dockerfile + registering in utils.sh.
• Infrastructure as Code: Terraform for cloud, Docker Compose for local.
• Configuration-driven: scenarios defined purely by .env files.
• Observability pipeline: JMX → Prometheus → Grafana + producer logs → Jupyter.