How mohameddiar/data-engineering-real-time-streaming-project works
Main Components
a) Data Sources:
• Simulated Glucose Devices (IoT)
• Device Health Monitoring
• MySQL Database (for CDC)
b) Ingestion Layer:
• Azure Event Hubs (8 different hubs)
• Debezium Server (for CDC)
c) Processing Layer:
• Azure Databricks (Stream Processing)
• Azure Functions (4 Function Apps)
• DuckDB (ETL Processing)
d) Storage Layer:
• Azure Database for MySQL
• Azure Cache for Redis
• Azure Blob Storage
e) Output Layer:
• Telegram Channels (5 different channels)
• Streamlit Dashboard
Connection Types
Use different line styles to represent different types of data flow:
Component Grouping
Components into logical sections:
a) Real-time Pipeline:
• Glucose Readings Stream
• Device Health Stream
• Alert Processing
b) Batch Pipeline:
• Daily ETL Process
• Monthly Trend Analysis
c) CDC Pipeline:
• Database Changes
• Cache Updates
CDC Processing
b) For each Event Hub, show:
• Input source
• Consumers
• Output destination
c) For Azure Functions, show:
• Trigger type (Event Hub/Timer)
• Processing logic
• Output destination
Additional Elements
Include:
• Processing intervals (e.g., "Daily at 2 AM")
• Component versions/specifications where relevant
Critical Flows to Highlight
a) Real-time alerts:
• Glucose readings → Event Hub → Databricks → Alert Event Hub → Function → Telegram
b) Daily ETL:
• MySQL → DuckDB → Blob Storage → Data Warehouse
c) CDC flow:
• MySQL → Debezium → Event Hub → Function → Redis Cache
Monitoring Points
Highlight:
• Application Insights integration
• Log Analytics
• Alert monitoring points