# jeanlouisboudart/kafka-producer-benchmark architecture

> Architecture diagram of the GitHub repository jeanlouisboudart/kafka-producer-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jeanlouisboudart/kafka-producer-benchmark
- Repository: https://github.com/jeanlouisboudart/kafka-producer-benchmark

## Overview

##### Project Type and Purpose
- This is an *open-source benchmarking framework* that orchestrates a Kafka cluster (locally via Docker Compose or in the cloud via Terraform/Kubernetes), runs a series of producer clients (Java, .NET, Go, Python, Rust), captures metrics, and produces comparative reports.

##### File-Structure Highlights
- **bench-initializer/** (Java): initializes topics and preloads data.
- **cloud/setup/**: Terraform files to provision Confluent Cloud + Kubernetes environment.
- **docker-compose*.yml**: multiple Kafka cluster topologies (Zookeeper vs KRaft, 1 vs 3 brokers).
- **\*-producer/**: one folder per client implementation, each containing a Dockerfile, language-specific code, config, and logging setup.
- **jmx-exporter/**, **prometheus/**, **grafana/**: monitoring stack.
- **run-scenarios*.sh**, **utils.sh**: orchestration scripts that discover all `scenario-*.env` files, build client images, launch scenarios, collect logs to `target/`.

##### Key Architectural Insights from README
- Benchmark scenarios are defined entirely by `.env` files.
- Clients are *pluggable*: you add a new folder with a Dockerfile and register it in `utils.sh`.
- Metrics flow: each producer logs to stdout → stored in `target/.../*.txt`; JMX-Exporter sidecars expose broker metrics → scraped by Prometheus → visualized in Grafana.
- Cloud mode uses Terraform to spin up Confluent Cloud resources and a K8s cluster, then reuses the same producer containers as Kubernetes Jobs/Pods.

##### Main Components & Their Interactions
a. **Orchestrator**
- *Local*: `docker-compose.yml` (and variants) + `run-scenarios.sh` / `run-scenario.sh` scripts.
- *Cloud*: Terraform (`cloud/setup/*.tf`) → Confluent Cloud + K8s cluster → `run-scenarios-cloud.sh`.

b. **Kafka Cluster**
- Brokers (Zookeeper-based or KRaft) running as Docker services or Cloud-managed brokers.

c. **JMX-Exporter**
- Attached to each broker (and optionally ZK) to expose JVM metrics on HTTP endpoints.

d. **Prometheus**
- Scrapes JMX-Exporter endpoints + can scrape custom metrics endpoints if producers expose any.

e. **Grafana**
- Dashboards provisioned under `grafana/provisioning/` that query Prometheus.

f. **Producer Clients**
- Multiple Docker images (java-producer, dotnet-producer, golang-producer, python-producer, rust-producer-sync).
- Launched in sequence or parallel by the orchestrator scripts.
- Each logs standardized metrics to stdout (Send rate, queue time, batch size, etc.).

g. **bench-initializer**
- Java utility that pre-creates topics based on env variables and optionally preloads messages.

h. **Reporting**
- After scenarios complete, raw logs reside in `target/...`.
- `report/benchmark-report.ipynb` and `extract_results.py` ingest these logs and produce charts.
- A Dockerized report container can build final PDF/HTML.

##### Architectural Patterns & Design Principles
- **Containerized micro-services**: each client & service in its own container.
- **Plug-in architecture**: new clients can be added by placing a folder + Dockerfile + registering in `utils.sh`.
- **Infrastructure as Code**: Terraform for cloud, Docker Compose for local.
- **Configuration-driven**: scenarios defined purely by `.env` files.
- **Observability pipeline**: JMX → Prometheus → Grafana + producer logs → Jupyter.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration
    subgraph "Orchestrator" 
        direction TB
        subgraph "Local Mode" 
            direction TB
            LS1["run-scenarios.sh"]:::orch
            LS2["run-scenario.sh"]:::orch
            LS3["utils.sh"]:::orch
            LC1["docker-compose.yml"]:::infra
            LC2["docker-compose-1-broker.yml"]:::infra
            LC3["docker-compose-3-brokers.yml"]:::infra
            LC4["docker-compose-kraft.yml"]:::infra
            LC5["docker-compose-kraft-3-brokers.yml"]:::infra
            LC6["docker-kraft-run.sh"]:::infra
        end
        subgraph "Cloud Mode"
            direction TB
            CS1["run-scenarios-cloud.sh"]:::orch
            CS2["run-scenario-cloud.sh"]:::orch
            CS3["destroy-cloud-env.sh"]:::orch
            TF1["clients.tf"]:::infra
            TF2["cluster.tf"]:::infra
            TF3["kubernetes.tf"]:::infra
            TF4["outputs.tf"]:::infra
            TF5["providers.tf"]:::infra
            TF6["variables.tf"]:::infra
            TF7["bash-to-tfvars.py"]:::infra
        end
    end

    %% Initialization
    BenchInit["bench-initializer"]:::orch

    %% Infrastructure - Kafka Cluster
    subgraph "Kafka Cluster" 
        direction TB
        Brokers["Brokers (ZK/KRaft)"]:::infra
    end

    %% Producer Clients
    subgraph "Producer Clients (Plug-in)" 
        direction TB
        Java["Java Producer"]:::producer
        Dotnet[".NET Producer"]:::producer
        GoLang["Go Producer"]:::producer
        Python["Python Producer"]:::producer
        Rust["Rust Producer"]:::producer
    end

    %% Observability / Monitoring
    subgraph "Monitoring Stack"
        direction TB
        JMX["JMX-Exporter"]:::monitor
        Prom["Prometheus"]:::monitor
        Graf["Grafana"]:::monitor
    end

    %% Reporting
    subgraph "Reporting"
        direction TB
        FileStore["target/... (logs)"]:::orch
        Report["Notebook & Scripts"]:::orch
    end

    %% Flows
    LS1 -->|"reads .env"| LC1
    LC1 -->|"brings up"| Brokers
    CS1 -->|"Terraform apply"| Brokers
    LS1 -->|"invoke"| BenchInit
    CS1 -->|"invoke"| BenchInit
    BenchInit -->|"create topics"| Brokers

    LS1 -->|"start clients"| Java
    LS1 --> Java
    LS1 --> Dotnet
    LS1 --> GoLang
    LS1 --> Python
    LS1 --> Rust

    CS1 --> Java
    CS1 --> Dotnet
    CS1 --> GoLang
    CS1 --> Python
    CS1 --> Rust

    Java -->|"produce to Kafka"| Brokers
    Dotnet --> Brokers
    GoLang --> Brokers
    Python --> Brokers
    Rust --> Brokers

    Brokers -->|"JVM metrics"| JMX
    JMX -->|"HTTP scrape"| Prom
    Prom -->|"query metrics"| Graf

    Java -->|"logs to stdout"| FileStore
    Dotnet --> FileStore
    GoLang --> FileStore
    Python --> FileStore
    Rust --> FileStore

    FileStore -->|"ingest"| Report

    %% Click Events
    click LS1 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/run-scenarios.sh"
    click LS2 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/run-scenario.sh"
    click LS3 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/utils.sh"
    click LC1 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-compose.yml"
    click LC2 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-compose-1-broker.yml"
    click LC3 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-compose-3-brokers.yml"
    click LC4 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-compose-kraft.yml"
    click LC5 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-compose-kraft-3-brokers.yml"
    click LC6 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/docker-kraft-run.sh"
    click CS1 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/run-scenarios-cloud.sh"
    click CS2 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/run-scenario-cloud.sh"
    click CS3 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/destroy-cloud-env.sh"
    click TF1 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/clients.tf"
    click TF2 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/cluster.tf"
    click TF3 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/kubernetes.tf"
    click TF4 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/outputs.tf"
    click TF5 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/providers.tf"
    click TF6 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/variables.tf"
    click TF7 "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/cloud/setup/bash-to-tfvars.py"
    click BenchInit "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/bench-initializer/"
    click Java "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/java-producer/"
    click Dotnet "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/dotnet-producer/"
    click GoLang "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/golang-producer/"
    click Python "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/python-producer/"
    click Rust "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/rust-producer-sync/"
    click JMX "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/jmx-exporter/"
    click Prom "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/prometheus/prometheus-config/"
    click Graf "https://github.com/jeanlouisboudart/kafka-producer-benchmark/tree/main/grafana/provisioning/"
    click Report "https://github.com/jeanlouisboudart/kafka-producer-benchmark/blob/main/report/benchmark-report.ipynb"

    %% Styles
    classDef infra fill:#dae8fc,stroke:#6c8ebf,color:#1f497d
    classDef producer fill:#d5e8d4,stroke:#82b366,color:#274e13
    classDef monitor fill:#ffe6cc,stroke:#d79b00,color:#7f6000
    classDef orch fill:#f2f2f2,stroke:#999999,color:#333333
    classDef reporter fill:#f2f2f2,stroke:#999999,color:#333333
```
