# bngiahuy/kafka-master architecture

> Architecture diagram of the GitHub repository bngiahuy/kafka-master, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bngiahuy/kafka-master
- Repository: https://github.com/bngiahuy/kafka-master

## Overview

Based on our analysis of the provided file tree and README, we’re looking at a distributed job-processing system—the “Kafka Master”—that serves as the coordinator (Master Node) for assigning work to various Worker Nodes. Communication across the system is handled via Kafka for asynchronous messaging, while Redis is used to keep track of worker statuses, job assignments, and system state. The project appears to be implemented as a Node.js application (located in the src/ directory) for the master services and exposes API endpoints to report system status and update configurations. There is also a dedicated Python component (in the python/ folder) that likely implements the workers (or a specific type of worker), each with it’s own Dockerfile and docker-compose configuration for containerized deployment.

##### Main Components
- Master Node: This is the core of the application. It hosts the main controller logic (src/index.js), configuration (inside src/configs/), API endpoints (routes and controllers), producer and consumer modules (src/producers & src/consumers), and additional utilities (src/utils and leaderElection).
- Worker Nodes: Represented by the Python-based code (python/worker.py) that likely runs in separate containers. They register with the master, listen for job assignments, and report back progress.
- Kafka: Acts as the messaging backbone with multiple topics (e.g., KAFKA_TOPIC_NAME_MASTER, KAFKA_TOPIC_NAME_WORKER, KAFKA_TOPIC_NAME_WORKER_FREE) used for distributing tasks and receiving updates.
- Redis: Used as a persistent and fast-access data store to track the system’s runtime state (worker statuses, job batches, leader election metadata, and file/chunk processing information).

##### Relationships and Interactions
- The Master Node reads input files and splits them into data batches. It then publishes these job assignments to the Kafka topic (master signal).
- Worker Nodes subscribe to a specific Kafka topic channel to pick up jobs. Upon receiving a job, they process the data and communicate their status (or completion) back through another Kafka topic.
- The Master continuously monitors worker progress using both Kafka messages and Redis, where it tracks registration, status updates, and job/chunk states.
- API endpoints exposed by the Master Node (documented in the README) allow external monitoring and adjustments (like updating batch sizes or Kafka partition information).

##### Architectural Patterns and Design Principles
- Service-Oriented/Modular Design: Notice the separation of components into logically distinct directories such as controllers, services, producers, and consumers.
- Event-Driven Architecture: The usage of Kafka topics clearly indicates an event-driven communication pattern, emphasizing decoupled and asynchronous processing.
- Containerization and Microservices: Docker and docker-compose files at both the project root and within the python subdirectory suggest that each component is containerized and potentially deployed as distinct services (for scaling or isolated development).

##### Incorporate Technologies, Frameworks, and Libraries
- Node.js/Express for the Master API and orchestration logic.
- Kafka for message brokering with configured topics.
- Redis as a fast key-value store for real-time state and status.
- Docker for containerization and easy deployment, including separate setups for Node.js and Python worker environments.
- The file structure also includes utility modules (logging, helper functions) that support the system’s maintenance and reliability.

##### Final Touches
- Components are clearly labeled.
- From Master → Kafka → Worker (job assignments).
- From Worker → Kafka → Master (heartbeat & processing updates).
- From Master → Redis and vice versa (status tracking, job chunk management).
- Differentiate between communication protocols and data flows where applicable (e.g., API calls vs. message queues).

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Master Node Subgraph
    subgraph "Master Node (Node.js)"
        M1["Main Controller (index.js)"]:::service
        subgraph "Configuration"
            M2["Kafka Config (kafkaConfig.js)"]:::service
            M3["Redis Config (redisConfig.js)"]:::service
        end
        subgraph "API Endpoints & Controllers"
            M4["Main Controller (mainController.js)"]:::api
            M5["Main Route (mainRoute.js)"]:::api
        end
        M6["Kafka Producer (kafkaProducer.js)"]:::messaging
        M7["Kafka Consumer (kafkaConsumer.js)"]:::messaging
        M8["Leader Election (leaderElection)"]:::service
        subgraph "Utility Modules"
            M9["Helper (helper.js)"]:::utility
            M10["Logger (logger.js)"]:::utility
            M11["Worker Manager (workerManager.js)"]:::utility
        end
    end

    %% Messaging System Subgraph
    subgraph "Messaging System"
        K["Kafka (Job & Status Messaging)"]:::messaging
    end

    %% Data Store Subgraph
    subgraph "Data Store"
        R["Redis"]:::db
    end

    %% Worker Node Subgraph
    subgraph "Worker Node (Python)"
        W["Python Worker (worker.py)"]:::service
        subgraph "Python Container Environment"
            CW1["Dockerfile"]:::container
            CW2["docker-compose.yml"]:::container
        end
    end

    %% Containerization for Master
    subgraph "Overall Project Containerization"
        CM1["Dockerfile"]:::container
        CM2["docker-compose.yml"]:::container
    end

    %% Relationships
    %% Master publishes jobs to Kafka
    M6 -->|"job_assignment"| K
    %% Kafka delivers jobs to Worker Node
    K -->|"job_delivery"| W
    %% Worker sends status update back to Kafka
    W -->|"status_update"| K
    %% Kafka passes status updates to Master's consumer
    K -->|"job_status"| M7
    %% Master monitors and updates status via Redis
    M1 -->|"status_tracking"| R
    R -->|"state_query"| M1

    %% Click Events for Master Node Components
    click M1 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/index.js"
    click M2 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/configs/kafkaConfig.js"
    click M3 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/configs/redisConfig.js"
    click M4 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/controllers/mainController.js"
    click M5 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/routes/mainRoute.js"
    click M6 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/producers/kafkaProducer.js"
    click M7 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/consumers/kafkaConsumer.js"
    click M8 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/leaderElection/index.js"
    click M9 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/utils/helper.js"
    click M10 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/utils/logger.js"
    click M11 "https://github.com/bngiahuy/kafka-master/blob/main-v1/src/utils/workerManager.js"

    %% Click Events for Worker Node and Containerization
    click W "https://github.com/bngiahuy/kafka-master/blob/main-v1/python/worker.py"
    click CW1 "https://github.com/bngiahuy/kafka-master/tree/main-v1/python/Dockerfile"
    click CW2 "https://github.com/bngiahuy/kafka-master/blob/main-v1/python/docker-compose.yml"
    click CM1 "https://github.com/bngiahuy/kafka-master/tree/main-v1/Dockerfile"
    click CM2 "https://github.com/bngiahuy/kafka-master/blob/main-v1/docker-compose.yml"

    %% Class Definitions with Colors
    classDef service fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef api fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef messaging fill:#FFC0CB,stroke:#000,stroke-width:2px;
    classDef db fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef utility fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef container fill:#FFA500,stroke:#000,stroke-width:2px;
```
