# debanjansaha-git/kafka-spark-cassandra architecture

> Architecture diagram of the GitHub repository debanjansaha-git/kafka-spark-cassandra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/debanjansaha-git/kafka-spark-cassandra
- Repository: https://github.com/debanjansaha-git/kafka-spark-cassandra

## Overview

##### Project Type and Purpose
- This is a real-time data streaming pipeline (data engineering project), not a typical web or mobile app.
- Purpose: simulate weather data → stream it via Kafka → transform with Spark Streaming → store in Cassandra → orchestrate with Airflow.

##### File Structure
- Top-level:
- docker-compose.yml – defines Kafka, Zookeeper, Spark, Cassandra, Airflow services
- requirements*.txt – Python dependencies
- spark_stream.py – Spark Streaming job
- kafka_stream.py – Kafka producer (weather data generator)
- dags/ – Airflow DAG definitions (weather_data_streaming.py)
- script/entrypoint.sh – container entrypoint for startup
- Patterns:
- “dags” indicates orchestration layer (Airflow)
- Separate scripts for producer (kafka_stream.py) and processor (spark_stream.py)

##### README Insights
- Technology stack: Kafka, Spark, Cassandra, Airflow, Docker, Faker
- Architecture section enumerates components and includes a reference image
- Flow: Data generation → Kafka topic → Spark Streaming → Cassandra

##### Main Components & Their Interactions
a. Weather Data Generator (Kafka Producer)
– Executes kafka_stream.py (orchestrated by Airflow PythonOperator)
– Uses Faker to emit JSON messages to Kafka topic “weather_data”
b. Apache Kafka Cluster
– Kafka Broker + Zookeeper (defined in Docker Compose)
– Receives and buffers incoming weather events
c. Apache Spark Streaming
– spark_stream.py reads from Kafka, flattens nested JSON
– Writes transformed records into Cassandra
d. Cassandra Database
– Keyspace: spark_streams, Table: weather_data
– Stores final records for analytics
e. Apache Airflow
– dag/weather_data_streaming.py orchestrates end-to-end pipeline
– Tasks: trigger producer, monitor stream, possibly kick off Spark job
f. Docker Compose
– Groups all services in isolated Docker network

##### Architectural Patterns and Principles
- Modular separation: producer, broker, consumer, database
- Event-driven streaming architecture
- Orchestration vs. processing vs. storage layers
- Resilience via Kafka buffering
- Scalability: Spark cluster + Cassandra distributed writes

##### Detail Level
- PythonOperator in the DAG triggering `kafka_stream.py`
- Both broker and Zookeeper if desired
- Topic name (`weather_data`)
- Keyspace/table in Cassandra
- Entrypoint.sh as container startup

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Docker Compose Network"
        direction TB
        subgraph "Orchestration Layer"
            direction TB
            AirflowUI["Airflow Web UI"]:::infra
            DAG["Weather Data Streaming DAG<br/>(PythonOperator)"]:::orchestration
            AirflowUI --> DAG
        end
        subgraph "Ingestion Layer"
            direction TB
            ProducerScript["Kafka Producer<br/>(weather_data_generator)"]:::streaming
            ZK["Zookeeper"]:::infra
            Kafka["Kafka Broker"]:::infra
            ProducerScript -->|"publish JSON"| KafkaTopic
            ZK --> Kafka
        end
        subgraph "Messaging"
            direction TB
            KafkaTopic["Topic: weather_data"]:::infra
        end
        subgraph "Processing Layer"
            direction TB
            SparkJob["Spark Streaming Job<br/>(flatten JSON)"]:::streaming
        end
        subgraph "Storage Layer"
            direction TB
            CassandraSrv["Cassandra Service"]:::storage
            CassandraTable["Keyspace: spark_streams<br/>Table: weather_data"]:::storage
            CassandraSrv --> CassandraTable
        end
    end

    DAG -->|"trigger"| ProducerScript
    Kafka -->|"buffer"| KafkaTopic
    KafkaTopic -->|"read & flatten"| SparkJob
    SparkJob -->|"write"| CassandraSrv

    subgraph "Project Artifacts"
        direction TB
        README["README.md"]:::infra
        ReqRoot["requirements.txt"]:::infra
        ReqLocal["requirements_local.txt"]:::infra
        Entrypoint["entrypoint.sh"]:::infra
        Assets["assets/"]:::infra
    end

    classDef orchestration fill:#D6E9FE,stroke:#333,stroke-width:1px
    classDef streaming fill:#FFDDAA,stroke:#333,stroke-width:1px
    classDef storage fill:#D4F7DC,stroke:#333,stroke-width:1px
    classDef infra fill:#E8E8E8,stroke:#333,stroke-width:1px

    click ProducerScript "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/dags/kafka_stream.py"
    click DAG "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/dags/kafka_stream.py"
    click Kafka "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/docker-compose.yml"
    click ZK "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/docker-compose.yml"
    click SparkJob "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/spark_stream.py"
    click CassandraSrv "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/docker-compose.yml"
    click Entrypoint "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/script/entrypoint.sh"
    click Assets "https://github.com/debanjansaha-git/kafka-spark-cassandra/tree/main/assets/"
    click ReqRoot "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/requirements.txt"
    click ReqLocal "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/requirements_local.txt"
    click README "https://github.com/debanjansaha-git/kafka-spark-cassandra/blob/main/README.md"
```
