# kaushal-dabhi/kafka_weather_monitoring_app architecture

> Architecture diagram of the GitHub repository kaushal-dabhi/kafka_weather_monitoring_app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaushal-dabhi/kafka_weather_monitoring_app
- Repository: https://github.com/kaushal-dabhi/kafka_weather_monitoring_app

## Overview

##### Project Type and Purpose
- This is a real‐time data streaming pipeline (producer–consumer) combined with a lightweight dashboard.

##### File Structure
- Top‐level directories:
- producer/weather_producer.py – Publishes to Kafka
- consumer/weather_consumer.py – Reads from Kafka, writes to PostgreSQL
- dashboard/app.py – Streamlit UI for real‐time visualization
- Configuration:
- docker-compose.yml – Orchestrates Kafka, Zookeeper, PostgreSQL, plus the three Python services
- requirements.txt – Lists Python dependencies (kafka-python, psycopg2, streamlit, etc.)

##### README Insights
- Confirms components: Kafka + Zookeeper cluster, Weather Producer, Weather Consumer (+ Postgres), Streamlit Dashboard
- Sketch of data flow:
Producer → Kafka Topic → Consumer → PostgreSQL (optional)
Producer & Consumer → Streamlit Dashboard (real‐time)

##### Main Components & Their Interactions
a. External OpenWeatherMap API
– The source of real‐time weather data
b. Weather Producer Service
– Polls API, serializes JSON, publishes onto a Kafka topic
c. Kafka Cluster (Broker + Zookeeper)
– Acts as the message bus (Pub/Sub)
d. Weather Consumer Service
– Subscribes to the same topic, processes messages, optionally persists to PostgreSQL
e. PostgreSQL Database (Optional Storage)
– Holds historical logs for analytics
f. Streamlit Dashboard
– Subscribes to Kafka topic (or polls Consumer/DB) for live updates

Interactions:

##### Architectural Patterns & Design Principles
- Event‐Driven / Pub-Sub Architecture (Producer–Broker–Consumer)
- Microservice‐style separation: each Python script runs in its own container
- Optional persistent storage for historical tracking
- Dashboard decoupled from processing logic

##### Technologies & Frameworks to Call Out
- kafka-python (client library)
- Docker & docker-compose (service orchestration)
- Streamlit (dashboard UI)
- psycopg2 / SQLAlchemy (PostgreSQL connectivity)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layer 1: Ingestion
    subgraph "Layer 1: Ingestion"
        API["OpenWeatherMap API"]:::external
    end

    %% Layer 2: Messaging Platform
    subgraph "Layer 2: Messaging Platform"
        direction LR
        ZK["Zookeeper"]:::service
        Broker["Kafka Broker"]:::service
    end

    %% Layer 3: Processing
    subgraph "Layer 3: Processing"
        Producer["Producer (Python + kafka-python)"]:::service
        Consumer["Consumer (Python + kafka-python, psycopg2)"]:::service
    end

    %% Layer 4: Storage
    subgraph "Layer 4: Storage"
        Postgres["PostgreSQL"]:::datastore
    end

    %% Layer 5: Presentation
    subgraph "Layer 5: Presentation"
        Dashboard["Streamlit Dashboard"]:::service
    end

    %% Documentation
    subgraph "Documentation"
        Requirements["requirements.txt"]:::docs
        README["README.md"]:::docs
    end

    %% Data Flow
    API -->|"Pull API Data"| Producer
    Producer -->|"Publish to topic: weather"| Broker
    ZK -->|"Coordinates cluster"| Broker
    Broker -->|"Subscribe to topic: weather"| Consumer
    Consumer -->|"Write data"| Postgres
    Producer -->|"Real-time data"| Dashboard
    Consumer -->|"Real-time data"| Dashboard
    Postgres -->|"Historical queries"| Dashboard

    %% Click Events
    click Producer "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/producer/weather_producer.py"
    click Consumer "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/consumer/weather_consumer.py"
    click Dashboard "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/dashboard/app.py"
    click Broker "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/docker-compose.yml"
    click Postgres "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/docker-compose.yml"
    click Requirements "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/requirements.txt"
    click README "https://github.com/kaushal-dabhi/kafka_weather_monitoring_app/blob/main/README.md"

    %% Styles
    classDef service fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20;
    classDef datastore fill:#FFF3E0,stroke:#EF6C00,color:#E65100;
    classDef external fill:#E0F7FA,stroke:#006064,color:#004D40;
    classDef docs fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C;
```
