# teslamotors/kafka-helmsman architecture

> Architecture diagram of the GitHub repository teslamotors/kafka-helmsman, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/teslamotors/kafka-helmsman
- Repository: https://github.com/teslamotors/kafka-helmsman

## Overview

##### Project Type & Purpose
- This is an open-source toolkit composed of several sub-projects (or “tools”) that automate specific Kafka management tasks such as consumer freshness tracking, topic enforcing, quota enforcement, certificate management, and cluster rolling.
- Its purpose is to reduce the manual toil involved in managing Kafka clusters while ensuring consistent, high-quality operational procedures.

##### Main Components
- Primary tool as a distinct module:
- Kafka Consumer Freshness Tracker (located under kafka_consumer_freshness_tracker)
- Kafka Topic Enforcer (under kafka_topic_enforcer)
- Kafka Quota Enforcer (under kafka_quota_enforcer)
- Kafka Roller (under kafka_roller)
- Kafka Certificates (under kafka_certificates)
- Kafka ACL Enforcer and the associated common functionalities (from kafka_acl_enforcer and kafka_enforcer_common)
- Recognize shared libraries and common components (e.g., kafka_enforcer_common) that multiple tools depend on.
- Both language-based subsystems:
- Java-based components, which use Bazel for build and test.
- Python-based components (e.g., Kafka Roller) built and tested using tox and custom scripts.

##### Relationships and Interactions
- Individual tool as a separate block. For example, display “Topic Enforcer”, “Quota Enforcer”, “Consumer Freshness Tracker”, “ACL Enforcer”, etc.
- Integration with external services: For instance, these tools interface with Kafka clusters (and likely ZooKeeper in the case of Kafka Roller) to perform their automation tasks.

##### Architectural Patterns and Design Principles
- The repository follows a modular structure where each tool is self-contained. Emphasize this component-based separation.
- The use of Bazel as a common build tool for Java demonstrates a hermetic, reproducible build process which is an important design decision.
- The separation of concerns is clear: each tool is focused on a specific task, and the existence of common libraries (e.g., kafka_enforcer_common) suggests reuse and shared logic.
- You might also show external dependency management (illustrated by the 3rdparty directory for Java libraries) as a separate layer.

##### Technology and Framework Dependencies
- Java 17 – for building the core tools; make a note about the Bazel build system and Bazelisk handling versioning.
- Python – for Kafka Roller (and any other Python scripts), tested and built using tox and custom shell scripts.
- External libraries (e.g., commonly used Kafka client libraries, logging frameworks like logback/log4j) that are present in the 3rdparty directory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Java Tools"
        KFCT["Kafka Consumer Freshness Tracker"]:::java
        KTEn["Kafka Topic Enforcer"]:::java
        KQEn["Kafka Quota Enforcer"]:::java
        KCert["Kafka Certificates"]:::java
        KACL["Kafka ACL Enforcer"]:::java
        Common["Common Enforcer Libraries"]:::shared
        Bazel["Bazel Build System (Java)"]:::build

        %% Bazel builds Java tools
        Bazel --> KFCT
        Bazel --> KTEn
        Bazel --> KQEn
        Bazel --> KCert
        Bazel --> KACL

        %% Java tools depend on common libraries
        KFCT -->|"uses"| Common
        KTEn -->|"uses"| Common
        KQEn -->|"uses"| Common
        KACL -->|"uses"| Common
        KCert -->|"may_use"| Common
    end

    subgraph "Python Tools"
        KRoller["Kafka Roller"]:::python
        Tox["tox & build_python.sh"]:::build
        Tox --> KRoller
    end

    KafkaCluster["Kafka Cluster (and optional ZooKeeper)"]:::external

    %% Integration of tools with Kafka Cluster
    KFCT -->|"integrates"| KafkaCluster
    KTEn -->|"integrates"| KafkaCluster
    KQEn -->|"integrates"| KafkaCluster
    KCert -->|"integrates"| KafkaCluster
    KACL -->|"integrates"| KafkaCluster
    KRoller -->|"integrates"| KafkaCluster

    T3rdParty["3rdparty Dependencies (Kafka Clients, Logging)"]:::build
    Bazel -->|"manages"| T3rdParty

    %% Click Events
    click KFCT "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_consumer_freshness_tracker/"
    click KTEn "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_topic_enforcer/"
    click KQEn "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_quota_enforcer/"
    click KCert "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_certificates/"
    click KACL "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_acl_enforcer/"
    click KRoller "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_roller/"
    click Common "https://github.com/teslamotors/kafka-helmsman/tree/master/kafka_enforcer_common/"

    %% Styles
    classDef java fill:#DDEEFF,stroke:#0077CC,stroke-width:2px,color:#000;
    classDef python fill:#FFEEDD,stroke:#CC7700,stroke-width:2px,color:#000;
    classDef external fill:#EEF2AA,stroke:#999900,stroke-width:2px,color:#000;
    classDef build fill:#E8E8E8,stroke:#666666,stroke-width:2px,color:#000;
    classDef shared fill:#FFDDFF,stroke:#990099,stroke-width:2px,color:#000;
```
