# rishamk/covid_symptoms_live_analysis architecture

> Architecture diagram of the GitHub repository rishamk/covid_symptoms_live_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishamk/covid_symptoms_live_analysis
- Repository: https://github.com/rishamk/covid_symptoms_live_analysis

## Overview

##### Project type and purpose
- This is a full-stack data-streaming and processing pipeline that simulates, ingests, stores, analyzes, and visualizes COVID-19 symptom data in real time.
- Key components: Kafka for streaming, HDFS for storage, Python scripts for production/consumption/analysis, Streamlit dashboard for visualization.

##### File structure
- Top-level files:
- fake_stream.py – Kafka producer (data simulation).
- consume_hdfs.py – Kafka consumer writing to HDFS.
- simple_analysis.py – Periodic read from HDFS and local plotting.
- dashboard.py – Streamlit-based real-time dashboard.
- docker-compose.yml – Orchestrates Zookeeper, Kafka, Hadoop NameNode/DataNode.
- requirements.txt – Python dependencies.
- README.md – Project overview.
- No separate “frontend”/“backend” folders; components are script-based services.

##### Extract architecture details from README
- Docker Compose spins up:
- Zookeeper
- Kafka broker
- Hadoop NameNode & DataNode
- Data flow:
[fake_stream.py] → Kafka topic “covid_symptoms” → [consume_hdfs.py] → HDFS → [simple_analysis.py / dashboard.py]

1. Data Source (covid_symptoms.csv)
2. Kafka Producer Service (fake_stream.py)
3. Apache Kafka Cluster (Broker + Zookeeper)
4. Kafka Consumer Service (consume_hdfs.py)
5. HDFS Cluster (NameNode + DataNode)
6. Analysis Service (simple_analysis.py)
7. Dashboard Service (dashboard.py via Streamlit)
8. User Interface (Web browser)
- Data Source → Producer
- Producer → Kafka topic
- Kafka → Consumer
- Consumer → HDFS (write)
- Analysis & Dashboard → HDFS (read)
- Dashboard → UI (HTTP)
c. Architectural patterns:
- Streaming pipeline with separate producer/consumer
- Batch-like storage in HDFS
- Real-time visualization on top of persisted data
- Containerized microservices via Docker Compose
d. Key technologies:
- kafka-python library (fake_stream.py, consume_hdfs.py)
- hdfs Python client
- pandas, matplotlib, numpy (analysis)
- Streamlit (dashboard)
- Docker Compose (infrastructure)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    DS["covid_symptoms.csv"]:::data

    %% Processing Services
    PS["Kafka Producer\n(fake_stream.py)"]:::proc
    CS["Kafka Consumer\n(consume_hdfs.py)"]:::proc
    AN["Analysis Service\n(simple_analysis.py)"]:::proc

    %% UI Layer
    DB["Dashboard Service\n(dashboard.py)"]:::ui
    UI["Web Browser"]:::ui

    %% Project Files
    subgraph "Project Files"
        direction TB
        RM["README.md"]:::doc
        RQ["requirements.txt"]:::doc
    end

    %% Infrastructure Boundary
    subgraph "Docker Compose Network"
        direction TB
        DC["docker-compose.yml"]:::infra
        subgraph "Kafka Cluster"
            direction TB
            ZK["Zookeeper\n(2181)"]:::stream
            KB["Kafka Broker\n(9092)\nTopic: covid_symptoms"]:::stream
        end
        subgraph "HDFS Cluster"
            direction TB
            NN["NameNode\n(9870)"]:::data
            DN["DataNode\n(9864)"]:::data
        end
    end

    %% Connections
    DS -->|reads CSV| PS
    PS -->|publishes| KB
    KB -->|streams| CS
    CS -->|writes| DN
    CS -->|writes| NN
    AN -->|reads| DN
    AN -->|reads| NN
    DB -->|reads| DN
    DB -->|reads| NN
    DB -->|HTTP| UI

    %% Click Events
    click PS "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/fake_stream.py"
    click CS "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/consume_hdfs.py"
    click AN "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/simple_analysis.py"
    click DB "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/dashboard.py"
    click DC "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/docker-compose.yml"
    click RM "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/README.md"
    click RQ "https://github.com/rishamk/covid_symptoms_live_analysis/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#ccffcc,stroke:#333;
    classDef stream fill:#cce5ff,stroke:#333;
    classDef proc fill:#ffe0b3,stroke:#333;
    classDef ui fill:#e5ccff,stroke:#333;
    classDef infra fill:#fff2cc,stroke:#333;
    classDef doc fill:#f2f2f2,stroke:#333;
```
