How rishamk/covid_symptoms_live_analysis works
Project type and purpose
• This is a full-stack data-streaming and processing pipeline that simulates, ingests, stores, analyzes, and visualizes COVID-19 symptom data in real time.
• Key components: Kafka for streaming, HDFS for storage, Python scripts for production/consumption/analysis, Streamlit dashboard for visualization.
File structure
• Top-level files:
• fake_stream.py – Kafka producer (data simulation).
• consume_hdfs.py – Kafka consumer writing to HDFS.
• simple_analysis.py – Periodic read from HDFS and local plotting.
• dashboard.py – Streamlit-based real-time dashboard.
• docker-compose.yml – Orchestrates Zookeeper, Kafka, Hadoop NameNode/DataNode.
• requirements.txt – Python dependencies.
• README.md – Project overview.
• No separate “frontend”/“backend” folders; components are script-based services.
Extract architecture details from README
• Docker Compose spins up:
• Zookeeper
• Kafka broker
• Hadoop NameNode & DataNode
• Data flow:
[fake_stream.py] → Kafka topic “covid_symptoms” → [consume_hdfs.py] → HDFS → [simple_analysis.py / dashboard.py]
1. Data Source (covid_symptoms.csv)
2. Kafka Producer Service (fake_stream.py)
3. Apache Kafka Cluster (Broker + Zookeeper)
4. Kafka Consumer Service (consume_hdfs.py)
5. HDFS Cluster (NameNode + DataNode)
6. Analysis Service (simple_analysis.py)
7. Dashboard Service (dashboard.py via Streamlit)
8. User Interface (Web browser)
• Data Source → Producer
• Producer → Kafka topic
• Kafka → Consumer
• Consumer → HDFS (write)
• Analysis & Dashboard → HDFS (read)
• Dashboard → UI (HTTP)
c. Architectural patterns:
• Streaming pipeline with separate producer/consumer
• Batch-like storage in HDFS
• Real-time visualization on top of persisted data
• Containerized microservices via Docker Compose
d. Key technologies:
• kafka-python library (fake_stream.py, consume_hdfs.py)
• hdfs Python client
• pandas, matplotlib, numpy (analysis)
• Streamlit (dashboard)
• Docker Compose (infrastructure)