How kaushal-dabhi/kafka_weather_monitoring_app works
Project Type and Purpose
• This is a real‐time data streaming pipeline (producer–consumer) combined with a lightweight dashboard.
File Structure
• Top‐level directories:
• producer/weather_producer.py – Publishes to Kafka
• consumer/weather_consumer.py – Reads from Kafka, writes to PostgreSQL
• dashboard/app.py – Streamlit UI for real‐time visualization
• Configuration:
• docker-compose.yml – Orchestrates Kafka, Zookeeper, PostgreSQL, plus the three Python services
• requirements.txt – Lists Python dependencies (kafka-python, psycopg2, streamlit, etc.)
README Insights
• Confirms components: Kafka + Zookeeper cluster, Weather Producer, Weather Consumer (+ Postgres), Streamlit Dashboard
• Sketch of data flow:
Producer → Kafka Topic → Consumer → PostgreSQL (optional)
Producer & Consumer → Streamlit Dashboard (real‐time)
Main Components & Their Interactions
a. External OpenWeatherMap API
– The source of real‐time weather data
b. Weather Producer Service
– Polls API, serializes JSON, publishes onto a Kafka topic
c. Kafka Cluster (Broker + Zookeeper)
– Acts as the message bus (Pub/Sub)
d. Weather Consumer Service
– Subscribes to the same topic, processes messages, optionally persists to PostgreSQL
e. PostgreSQL Database (Optional Storage)
– Holds historical logs for analytics
f. Streamlit Dashboard
– Subscribes to Kafka topic (or polls Consumer/DB) for live updates
Interactions:
Architectural Patterns & Design Principles
• Event‐Driven / Pub-Sub Architecture (Producer–Broker–Consumer)
• Microservice‐style separation: each Python script runs in its own container
• Optional persistent storage for historical tracking
• Dashboard decoupled from processing logic
Technologies & Frameworks to Call Out
• kafka-python (client library)
• Docker & docker-compose (service orchestration)
• Streamlit (dashboard UI)
• psycopg2 / SQLAlchemy (PostgreSQL connectivity)