# alexklein/kafka_data_processing architecture

> Architecture diagram of the GitHub repository alexklein/kafka_data_processing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alexklein/kafka_data_processing
- Repository: https://github.com/alexklein/kafka_data_processing

## Overview

##### Project type and purpose
- This is a real-time streaming data‐processing pipeline. It uses Apache Kafka as a message bus, a Python “Producer” service to generate simulated stock quotes, and a Python “Consumer” service that both writes raw data into PostgreSQL and processes it with Apache Spark. Docker Compose ties everything together.

##### File structure
- Top level:
- project/docker‐compose.yml – orchestrates all services (Kafka, Zookeeper, Postgres, Producer, Consumer, Spark).
- project/producer – Dockerfile + producer.py (generates stock data, publishes to Kafka topic “simulated_stock_prices”).
- project/consumer – Dockerfile + consumer.py variants (consumer_pg.py for direct Postgres writes; consumer_spark.py for Spark streaming).
- readme.md – describes components, build/run steps, and monitoring.

##### README for additional insights
- It names all major technologies (Kafka, Spark, PostgreSQL).
- It explains that the consumer forks into two threads: one to load raw data into Postgres, one to process via Spark with multiple sinks (console, CSV, Postgres).

##### Main components
a. Producer Service (Docker container running producer.py)
b. Kafka Cluster (Zookeeper + single Kafka broker, both in containers)
c. Consumer Service (Docker container running two consumer threads)
– Postgres Consumer
– Spark Consumer (pyspark streaming job)
d. Apache Spark (embedded or standalone cluster; exposes Spark UI)
e. PostgreSQL Database (data warehouse)
f. Optional: Monitoring UIs (Kafka Manager, Spark UI)

##### Relationships & data flows
1) Producer → Kafka topic “simulated_stock_prices”
2) Kafka → Consumer threads (two parallel pipelines)
- Pipeline A: write raw messages → PostgreSQL
- Pipeline B: ingest into Spark → process → output to console/CSV or PostgreSQL
3) Docker Compose network connects all containers.

##### Architectural patterns & design principles
- Microservices: decoupled Producer and Consumer services
- Event-driven streaming: Kafka topic as backbone
- Dual‐sink consumer: raw storage + analytic processing
- Containerization: Docker + Compose for reproducible environments

##### Key technologies & frameworks
- Python + kafka-python for Producer/Consumer
- Apache Kafka & Zookeeper
- Apache Spark (Structured Streaming)
- PostgreSQL
- Docker & Docker Compose

– From Kafka → Consumer/Postgres & Kafka → Spark
– From Spark → Postgres/Console/CSV
– Containers/services (e.g., green rectangles)
– Analytics engine (yellow hexagon or parallelogram for Spark)
- Networking overlay: show Docker network connecting all containers.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Docker Compose" 
        direction TB
        Producer["Producer Service"]:::service
        Zookeeper["Zookeeper"]:::broker
        Kafka["Kafka Broker\n(9092)"]:::broker
        Consumer["Consumer Service"]:::service
        subgraph "Consumer Pipelines"
            direction TB
            PostgresConsumer["Postgres Consumer"]:::service
            SparkConsumer["Spark Consumer"]:::service
        end
        Spark["Apache Spark\n(UI:8080)"]:::spark
        PostgresDB["PostgreSQL DB\n(5432)"]:::db
        README["Documentation\n(readme.md)"]:::doc
    end

    Producer -->|"simulated_stock_prices"| Kafka
    Kafka -->|"raw messages"| PostgresConsumer
    Kafka -->|"stream data"| SparkConsumer
    PostgresConsumer -->|"insert raw data"| PostgresDB
    SparkConsumer -->|"Structured Streaming"| Spark
    Spark -->|"write analytics"| PostgresDB
    Spark -->|"output to console"| Console["Console Sink"]:::service
    Spark -->|"write CSV"| CSV["CSV Sink"]:::service

    click Producer "https://github.com/alexklein/kafka_data_processing/blob/master/project/producer/producer.py"
    click PostgresConsumer "https://github.com/alexklein/kafka_data_processing/blob/master/project/consumer/consumer_pg.py"
    click SparkConsumer "https://github.com/alexklein/kafka_data_processing/blob/master/project/consumer/consumer_spark.py"
    click README "https://github.com/alexklein/kafka_data_processing/blob/master/readme.md"
    click DockerCompose "https://github.com/alexklein/kafka_data_processing/blob/master/project/docker-compose.yml"

    classDef service fill:#9f6,stroke:#333,stroke-width:1px
    classDef broker fill:#f96,stroke:#333,stroke-width:1px
    classDef db fill:#69f,stroke:#333,stroke-width:1px
    classDef spark fill:#fc6,stroke:#333,stroke-width:1px
    classDef doc fill:#ccc,stroke:#333,stroke-width:1px
```
