# mohameddiar/data-engineering-real-time-streaming-project architecture

> Architecture diagram of the GitHub repository mohameddiar/data-engineering-real-time-streaming-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohameddiar/data-engineering-real-time-streaming-project
- Repository: https://github.com/mohameddiar/data-engineering-real-time-streaming-project

## Overview

##### Main Components

a) Data Sources:
- Simulated Glucose Devices (IoT)
- Device Health Monitoring
- MySQL Database (for CDC)

b) Ingestion Layer:
- Azure Event Hubs (8 different hubs)
- Debezium Server (for CDC)

c) Processing Layer:
- Azure Databricks (Stream Processing)
- Azure Functions (4 Function Apps)
- DuckDB (ETL Processing)

d) Storage Layer:
- Azure Database for MySQL
- Azure Cache for Redis
- Azure Blob Storage

e) Output Layer:
- Telegram Channels (5 different channels)
- Streamlit Dashboard

##### Connection Types

Use different line styles to represent different types of data flow:

##### Component Grouping

Components into logical sections:
a) Real-time Pipeline:
- Glucose Readings Stream
- Device Health Stream
- Alert Processing

b) Batch Pipeline:
- Daily ETL Process
- Monthly Trend Analysis

c) CDC Pipeline:
- Database Changes
- Cache Updates

##### CDC Processing

b) For each Event Hub, show:
- Input source
- Consumers
- Output destination

c) For Azure Functions, show:
- Trigger type (Event Hub/Timer)
- Processing logic
- Output destination

##### Additional Elements

Include:
- Processing intervals (e.g., "Daily at 2 AM")
- Component versions/specifications where relevant

##### Critical Flows to Highlight

a) Real-time alerts:
- Glucose readings → Event Hub → Databricks → Alert Event Hub → Function → Telegram

b) Daily ETL:
- MySQL → DuckDB → Blob Storage → Data Warehouse

c) CDC flow:
- MySQL → Debezium → Event Hub → Function → Redis Cache

##### Monitoring Points

Highlight:
- Application Insights integration
- Log Analytics
- Alert monitoring points

## Diagram (Mermaid)

```mermaid
graph TB
    classDef azure fill:#0078D4,color:white
    classDef source fill:#4CAF50,color:white
    classDef processing fill:#FF9800,color:white
    classDef storage fill:#FFD700,color:black
    classDef output fill:#9C27B0,color:white
    classDef alert fill:#F44336,color:white

    subgraph "Data Sources"
        GD["Simulated Glucose Devices"]:::source
        DH["Device Health Monitoring"]:::source
        MSQL["MySQL Database"]:::storage
    end

    subgraph "Ingestion Layer"
        EH["Azure Event Hubs (8)"]:::azure
        DS["Debezium Server"]:::processing
    end

    subgraph "Processing Layer"
        DB["Azure Databricks"]:::azure
        AF["Azure Functions"]:::azure
        DUCK["DuckDB ETL"]:::processing
    end

    subgraph "Storage Layer"
        AZDB["Azure Database for MySQL"]:::azure
        RC["Azure Redis Cache"]:::azure
        BS["Azure Blob Storage"]:::azure
    end

    subgraph "Output Layer"
        TC["Telegram Channels"]:::output
        SD["Streamlit Dashboard"]:::output
    end

    GD --> EH
    DH --> EH
    MSQL --> DS
    DS --> EH
    EH --> DB
    EH --> AF
    DB --> AF
    AF --> TC
    AF --> RC
    AF --> AZDB
    AF --> BS
    DUCK --> BS
    RC --> SD
    AZDB --> SD

    %% Click Events
    click GD "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/src/Simulation/device_data_generator.py"
    click DB "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/Azure Databricks Notebooks/Glucose Monitoring Streaming Business Logic.ipynb"
    click AF "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/tree/master/Azure Functions/"
    click DUCK "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/Azure Functions/ETLDuckdb/glucose_aggregate.sql"
    click RC "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/tree/master/terraform_workspace/modules/azure_redis_cache/"
    click TC "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/src/Telegram_Setup/create_channels.py"
    click SD "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/real-time-live-streamlit-dashboard-python/app.py"
    click EH "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/tree/master/terraform_workspace/modules/azure_event_hub_namespace/"
    click DS "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/blob/master/terraform_workspace/modules/azure_vm/debezium_setup.sh"
    click AZDB "https://github.com/MohamedDiar/Data-Engineering-Real-Time-Streaming-Project/tree/master/terraform_workspace/modules/azure_sql_database/"

    %% Legend
    subgraph Legend
        L1["Azure Services"]:::azure
        L2["Data Sources"]:::source
        L3["Processing"]:::processing
        L4["Storage"]:::storage
        L5["Output"]:::output
    end
```
