# sa4s-serc/kafka_demo architecture

> Architecture diagram of the GitHub repository sa4s-serc/kafka_demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sa4s-serc/kafka_demo
- Repository: https://github.com/sa4s-serc/kafka_demo

## Overview

##### Project Type and Purpose
- This is a demo “kafka_demo” repository illustrating two styles of event-driven ML pipelines with Apache Kafka and Docker Compose:
a. Choreography: a simple producer → Kafka → consumer flow that applies a pre-built ML model.
b. Orchestration: a multi-step pipeline (data generation, training, model‐update, prediction) where an orchestrator service steers the flow.

##### Top-Level File Structure & Key Directories
- Choreography/
- producer/ (producer.py, Dockerfile)
- consumer/ (consumer.py, model.py, Dockerfile)
- docker-compose.yml (starts Zookeeper + Kafka broker)
- Orchestration/
- data_generator.py, trainer.py, orchestrator.py, predictor.py
- kd.py (Kafka topic definitions), logger_config.py
- cricket_model.pkl (trained artifact), docker-compose.yml
- README.md (usage and environment instructions)

##### Main Architectural Components
- Kafka Platform (Zookeeper + Broker)
- Choreography Services
- Producer Service (publishes raw events)
- Consumer Service (loads model.py, consumes events, does inference)
- Orchestration Services
- Data-Generator Service (publishes synthetic data events)
- Trainer Service (consumes data, trains model, emits model-update events)
- Orchestrator Service (controls the sequence: kicks off training, awaits completion, triggers prediction)
- Predictor Service (loads latest model artifact, consumes data, produces predictions)

##### Component Interactions & Data Flow
- Choreography Style: Producer → “raw‐data” topic → Consumer → (offline) prediction output
- Orchestration Style:
- Orchestrator → control topic → Trainer
- Data-Generator → “raw‐data” topic → Trainer & Predictor
- Trainer → model artifacts (cricket_model.pkl) + “model‐update” topic → Predictor
- Predictor → “prediction‐results” topic or logs

##### Architectural Patterns & Design Principles
- Event-driven microservices communicating via Kafka topics
- Containerized deployment via Docker Compose (each service in its own container)
- Two coordination styles:
- Choreography: each service reacts independently to events
- Orchestration: a central coordinator (Orchestrator) issues control-flow messages
- Separation of concerns: data generation, model training, inference, and control logic in distinct services

##### Relevant Technologies & Frameworks
- Python + kafka-python library (v2.0.2)
- Docker & Docker Compose for multi-container orchestration
- Pickle for model persistence (cricket_model.pkl)
- logger_config.py for structured logging across services
- kd.py for centralized topic‐name definitions (ensures uniform topic references)

##### Final Recommendations
- Both the choreography (event-driven only) and orchestration (central controller) flows to demonstrate the two architectural styles in one unified document.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Choreography Panel
    subgraph "Choreography (Docker Compose)"
        direction TB
        CH_Producer["Producer Service"]:::data
        CH_Kafka["Kafka Cluster\n(Zookeeper + Broker)\nTopics: raw-data"]:::kafka
        CH_Consumer["Consumer Service"]:::data
        CH_ModelC["Model (model.py)"]:::storage
        %% Connections
        CH_Producer -->|"publish raw-data"| CH_Kafka
        CH_Kafka -->|"consume raw-data"| CH_Consumer
        CH_Consumer -->|"loads model"| CH_ModelC
    end

    %% Orchestration Panel
    subgraph "Orchestration (Docker Compose)"
        direction TB
        OR_Orchestrator["Orchestrator Service"]:::control
        OR_ControlTopic["control topic"]:::kafka
        OR_DataGen["Data-Generator Service"]:::data
        OR_Kafka2["Kafka Cluster\n(Zookeeper + Broker)\nTopics: raw-data, model-update, prediction-results"]:::kafka
        OR_Trainer["Trainer Service"]:::control
        OR_ModelStore["Model Artifact\n(cricket_model.pkl)"]:::storage
        OR_Predictor["Predictor Service"]:::data
        %% Connections
        OR_Orchestrator -->|"trigger training"| OR_ControlTopic
        OR_ControlTopic -->|"start job"| OR_Trainer
        OR_DataGen -->|"publish raw-data"| OR_Kafka2
        OR_Kafka2 -->|"consume raw-data"| OR_Trainer
        OR_Trainer -->|"emit model-update"| OR_Kafka2
        OR_Trainer -->|"store model"| OR_ModelStore
        OR_Kafka2 -->|"consume model-update"| OR_Predictor
        OR_ModelStore -->|"load latest model"| OR_Predictor
        OR_Predictor -->|"publish predictions"| OR_Kafka2
    end

    %% Click Events - Choreography
    click CH_Producer "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/producer/producer.py"
    click CH_Consumer "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/consumer/consumer.py"
    click CH_ModelC "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/consumer/model.py"
    click CH_Producer "https://github.com/sa4s-serc/kafka_demo/tree/main/Choreography/producer/Dockerfile"
    click CH_Producer "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/producer/requirements.txt"
    click CH_Consumer "https://github.com/sa4s-serc/kafka_demo/tree/main/Choreography/consumer/Dockerfile"
    click CH_Consumer "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/consumer/requirements.txt"
    click CH_Kafka "https://github.com/sa4s-serc/kafka_demo/blob/main/Choreography/docker-compose.yml"

    %% Click Events - Orchestration
    click OR_Orchestrator "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/orchestrator.py"
    click OR_DataGen "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/data_generator.py"
    click OR_Trainer "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/trainer.py"
    click OR_Predictor "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/predictor.py"
    click OR_Kafka2 "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/docker-compose.yml"
    click OR_ModelStore "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/cricket_model.pkl"
    click OR_ControlTopic "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/kd.py"
    click OR_Orchestrator "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/logger_config.py"
    click OR_Trainer "https://github.com/sa4s-serc/kafka_demo/blob/main/Orchestration/requirements.txt"

    %% Styles
    classDef data fill:#AED6F1,stroke:#1B4F72,color:#0B3C5D;
    classDef control fill:#ABEBC6,stroke:#196F3D,color:#145A32;
    classDef kafka fill:#D5D8DC,stroke:#566573,color:#2C3E50;
    classDef storage fill:#F9E79F,stroke:#B7950B,color:#7D6608;
```
