# pritomdas/real-time-streaming-data-pipeline-and-dashboard architecture

> Architecture diagram of the GitHub repository pritomdas/real-time-streaming-data-pipeline-and-dashboard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard
- Repository: https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard

## Overview

##### Project Type & Purpose
- This is a full-stack real-time data pipeline project that integrates a simulation of server status data, multiple streaming data pipelines (one implemented in Scala and another in PySpark), and a dashboard built with Django/Flexmonster for visualization.
- The system utilizes technologies like Apache Kafka, Apache Spark (both Scala and PySpark based), Hadoop, ZooKeeper, PostgreSQL, and Docker for containerizing and running clusters.

##### File Structure
- There are three main parts:
- “data_center_server_live_status”: Contains Python scripts to simulate live server status data (and a Kafka consumer demo).
- “datamaking_real_time_data_pipeline (PySpark)” and “datamaking_streaming_data_pipeline (Scala)”: They include streaming pipeline implementations. The Scala project has the typical Scala/Java project directories (src, project, libraries, target) and uses build.sbt.
- “server_status_monitoring”: A Django project (with dashboard, models, views, templates, manage.py) that consumes data and shows a live dashboard.
- The presence of Docker setup scripts (mentioned in README) and dependencies in the Scala project (via external libraries) indicate that the system is deployed on a Docker environment incorporating multiple clusters.

##### Important Components to Highlight
a. Data Sources & Ingestion:
- Data Simulators: The Python simulator in “data_center_server_live_status” generates live server status events.
- Kafka Producer: The simulation data likely gets pushed into Kafka topics (as indicated by the Kafka consumer and Docker setup steps).

b. Messaging & Streaming:
- Kafka Broker: Use one or more Docker containers acting as the Kafka broker with its associated ZooKeeper container.
- Streaming Pipelines: Two distinct streaming modules are provided:
- Scala-based Spark Structured Streaming pipeline from “datamaking_streaming_data_pipeline (Scala)”
- PySpark-based streaming pipeline in “datamaking_real_time_data_pipeline (PySpark)”
These pipelines subscribe to Kafka topics, process the streaming data via Spark (possibly on an Apache Hadoop cluster), and then persist results.

c. Storage and Processing:
- Hadoop Cluster: Acts as scalable storage and processing framework; clusters are built via Docker.
- Apache Spark: The streaming computations (both Scala and PySpark) are executed here.
- PostgreSQL Database: The processed results (server events) are stored in a PostgreSQL database for use by the dashboard.

d. User Interface & Visualization:
- Django Web Application: Located in the “server_status_monitoring” folder, it handles dashboard routing, data fetch, and displays real-time visualizations using Flexmonster.

e. Docker & Cluster Management:
- Docker Containers: Overall, the infrastructure (Kafka, ZooKeeper, Hadoop, Spark, PostgreSQL) is managed and deployed within Docker.

##### Relationships and Data Flow
- Data Flow starts with the Python-based event simulator that sends server status events into Kafka.
- Kafka, managed in a Docker-based cluster along with ZooKeeper, acts as a distributed message queue.
- The streaming pipelines (Scala and PySpark) consume this data from Kafka. They process data in real-time using Spark and (optionally) store intermediate results into the Hadoop environment when needed.
- Post-processed data is then inserted into a PostgreSQL database.
- The Django dashboard application queries PostgreSQL and uses Flexmonster for data visualization to show a live dashboard.

##### Architectural Patterns & Design Principles
- Microservices/Containerization: Each major component (data simulator, Kafka, Spark Streaming, PostgreSQL, Django dashboard) is decoupled and possibly run as separate Docker containers.
- Real-Time Streaming: Use of Kafka and Spark Streaming indicates an event-driven, real-time processing architecture.
- Separation of Concerns: The data ingestion, processing, storage, and visualization are clearly separated.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Generation Layer
    subgraph "Data Generation"
        DS["Data Simulator – Python"]:::data_ingestion
        KCD["Kafka Consumer Demo"]:::data_ingestion
    end

    %% Docker Environment Container
    subgraph "Docker Environment"
        %% Messaging Layer inside Docker
        subgraph "Messaging Layer"
            KB["Kafka Broker"]:::messaging
            ZK["ZooKeeper"]:::messaging
        end
        %% Streaming Pipelines inside Docker
        subgraph "Streaming Pipelines"
            SPS["Spark Streaming (Scala)"]:::streaming
            SPY["Spark Streaming (PySpark)"]:::streaming
        end
        HC["Hadoop Cluster"]:::storage
        PG["PostgreSQL Database"]:::storage
    end

    %% User Interface Layer
    subgraph "User Interface"
        DD["Django Dashboard Application"]:::dashboard
    end

    %% Data Flow Connections
    DS -->|"pushes_data"| KB
    KCD -.->|"demo_read"| KB
    KB -->|"streams_data"| SPS
    KB -->|"streams_data"| SPY
    SPS -->|"writes_results"| PG
    SPY -->|"writes_results"| PG
    SPS -.->|"uses_storage"| HC
    SPY -.->|"uses_storage"| HC
    PG -->|"serves_data"| DD

    %% Click Events
    click DS "https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard/blob/main/data_center_server_live_status/data_center_server_live_status_simulator.py"
    click KCD "https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard/blob/main/data_center_server_live_status/kafka_consumer_demo.py"
    click SPY "https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard/blob/main/datamaking_real_time_data_pipeline (PySpark)/real_time_streaming_data_pipeline.py"
    click SPS "https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard/blob/main/datamaking_streaming_data_pipeline (Scala)/src/main/scala/streaming_data_pipeline.scala"
    click DD "https://github.com/pritomdas/real-time-streaming-data-pipeline-and-dashboard/tree/main/server_status_monitoring"

    %% Styles for Color Coding
    classDef data_ingestion fill:#a8e6a1,stroke:#333,stroke-width:2px;
    classDef messaging fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef streaming fill:#aed6f1,stroke:#333,stroke-width:2px;
    classDef storage fill:#f5cba7,stroke:#333,stroke-width:2px;
    classDef dashboard fill:#d2b4de,stroke:#333,stroke-width:2px;
```
