# mjngxwnj/traffic-and-weather-data-streaming-pipeline architecture

> Architecture diagram of the GitHub repository mjngxwnj/traffic-and-weather-data-streaming-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline
- Repository: https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline

## Overview

##### Project Type and Purpose
- This is a full‐stack streaming data pipeline for near‐real‐time processing of traffic camera images and weather data.
- It ingests images from traffic cameras (YOLOv8 via images_processing/), fetches weather data via OpenWeather API (weather_fetching/), streams data through Kafka (kafka_stream.py), processes it in Spark Streaming and Spark Batch (spark_script/), stores results in Cassandra and HDFS, and serves metrics via a Streamlit dashboard (application/).

##### File Structure Analysis
- Top-Level Components:
- application/ – Streamlit front-end and back-end, Dockerfile, requirements.
- images_processing/ – YOLOv8 image ingestion and vehicle count logic.
- weather_fetching/ – OpenWeather API client.
- spark_script/ – batch_job.py (daily batch logic), stream_job.py (real-time Spark Streaming), utils_spark.py.
- kafka_stream.py – Kafka producer for traffic and weather events.
- cql/ – Cassandra keyspace and table definitions.
- docker-compose.yaml & hadoop.env – container orchestration and Hadoop config.
- run_stream_job.sh, run_batch_job.sh – orchestration scripts.

##### Key Architectural Patterns
- Lambda (Hybrid) Architecture:
- Stream Layer – Spark Streaming consumes Kafka topics, writes to Cassandra (low latency).
- Batch Layer – Spark batch jobs consume historical Kafka data, write partitions to HDFS & aggregated results to Cassandra.
- Serving Layer – Cassandra serves combined real-time and batch data for low-latency queries; Streamlit provides interactive UI.
- Micro-services & Containerization – each major component runs in its own Docker container via docker-compose.
- Message Queue – Kafka for decoupling ingestion from processing, enabling fault tolerance and multiple consumers.

##### Main Components & Interactions
A. Data Ingestion
- Traffic Camera Module (images_processing/images_processor.py)
- Weather Fetching Module (weather_fetching/weather_utils.py)
→ produce events to Kafka topics (traffic_topic, weather_topic).
B. Kafka Cluster
- Brokers and Zookeeper (defined in docker-compose.yaml)
→ buffer and replicate incoming events.
C. Stream Processing
- Spark Streaming Job (spark_script/stream_job.py, run_stream_job.sh)
→ reads from Kafka, transforms data (density, humidex, etc.), writes to Cassandra stream_ tables.
D. Batch Processing
- Spark Batch Job (spark_script/batch_job.py, run_batch_job.sh)
→ scheduled daily via cron, reads historical events from Kafka or HDFS, writes raw to HDFS (partitioned by year/month/day), aggregates, writes to Cassandra batch_ tables.
E. Storage
- HDFS – raw and partitioned historical data (configured by hadoop.env).
- Cassandra – real-time and aggregated tables (cql/keyspace_table.cql).
F. Serving Layer
- Streamlit Application (application/main.py, streamlit_backend.py, streamlit_frontend.py)
→ queries Cassandra, renders dashboards.
- Dockerfile in application/ for containerizing the UI.

- Component names matching folder/script names (e.g., kafka_stream.py, stream_job.py).
- Environment configuration (docker-compose, hadoop.env).
- Data Ingestion (blue),
- Messaging (orange),
- Processing (green),
- Storage (purple),
- Serving (gray).
- Rectangles for services/apps,

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion"
        IM["Traffic Camera Module<br/>images_processing/images_processor.py"]:::ingestion
        WM["Weather Fetching Module<br/>weather_fetching/weather_utils.py"]:::ingestion
    end

    IM -->|traffic_events| KP["Kafka Producer<br/>kafka_stream.py"]:::messaging
    WM -->|weather_events| KP

    subgraph "Kafka Cluster (Docker)"
        BROKERS["Kafka Brokers"]:::messaging
        ZK["Zookeeper"]:::messaging
    end

    KP --> BROKERS
    BROKERS --> ZK

    subgraph "Stream Layer (Spark Streaming)"
        SS["Spark Streaming Job<br/>spark_script/stream_job.py"]:::processing
        OR_SS["Orchestration Script<br/>run_stream_job.sh"]:::processing
    end

    BROKERS -->|traffic_events & weather_events| SS
    OR_SS --> SS
    SS -->|writes to stream_ tables| CAS_STREAM["Cassandra Stream Tables"]:::storage

    subgraph "Batch Layer (Spark Batch)"
        SB["Spark Batch Job<br/>spark_script/batch_job.py"]:::processing
        UT["Spark Batch Utilities<br/>spark_script/utils_spark.py"]:::processing
        OR_SB["Orchestration Script<br/>run_batch_job.sh"]:::processing
    end

    BROKERS -->|historical_events| SB
    SB --> HDFS["HDFS Raw Data"]:::storage
    SB --> UT
    UT --> SB
    OR_SB --> SB
    SB -->|aggregated results| CAS_BATCH["Cassandra Batch Tables"]:::storage

    subgraph "Storage"
        HDFS
        CAS_STREAM
        CAS_BATCH
    end

    subgraph "Serving Layer"
        ST_APP["Streamlit App<br/>application/main.py"]:::serving
        ST_BE["Streamlit Backend<br/>application/streamlit_backend.py"]:::serving
        ST_FE["Streamlit Frontend<br/>application/streamlit_frontend.py"]:::serving
        DOCKER_APP["Dockerfile<br/>application/dockerfile"]:::serving
    end

    CAS_STREAM --> ST_BE
    CAS_BATCH --> ST_BE
    ST_BE --> ST_FE
    ST_FE --> ST_APP

    classDef ingestion fill:#add8e6,stroke:#000;
    classDef messaging fill:#ffa500,stroke:#000;
    classDef processing fill:#90ee90,stroke:#000;
    classDef storage fill:#dda0dd,stroke:#000;
    classDef serving fill:#d3d3d3,stroke:#000;

    click IM "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/images_processing/images_processor.py"
    click WM "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/weather_fetching/weather_utils.py"
    click KP "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/kafka_stream.py"
    click BROKERS "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/docker-compose.yaml"
    click ZK "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/docker-compose.yaml"
    click SS "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/spark_script/stream_job.py"
    click OR_SS "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/run_stream_job.sh"
    click SB "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/spark_script/batch_job.py"
    click UT "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/spark_script/utils_spark.py"
    click OR_SB "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/run_batch_job.sh"
    click CAS_STREAM "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/cql/keyspace_table.cql"
    click CAS_BATCH "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/cql/keyspace_table.cql"
    click HDFS "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/hadoop.env"
    click ST_APP "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/application/main.py"
    click ST_BE "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/application/streamlit_backend.py"
    click ST_FE "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/blob/main/application/streamlit_frontend.py"
    click DOCKER_APP "https://github.com/mjngxwnj/traffic-and-weather-data-streaming-pipeline/tree/main/application/dockerfile"
```
