How aymane-maghouti/real-time-data-pipeline-using-kafka works
This repository monitors local system performance through a threaded Python pipeline: it collects metrics, publishes them to Kafka, consumes and stores them in SQL Server, and presents them in a Power BI dashboard. The principal runtime stages are implemented in Main/data_pipeline.py, Main/producer.py, and Main/consumer.py. The README documents SQL Server setup and dashboard connectivity; the dashboard’s internal implementation is not shown, so its details are not expanded here.
Main entry points: data_pipeline.py Pipeline Orchestrator · consumer.py Kafka Consumer · producer.py Kafka Producer · sql-scripts.sql SQL Schema Setup
Components 8
Metric Runtime
- Pipeline Orchestrator
Main/data_pipeline.py
Kafka Streaming
- Kafka Producer
Main/producer.py - Kafka Broker
- Kafka Consumer
Main/consumer.py
SQL Storage
- SQL Schema Setup
Main/sql-scripts.sql - SQL Server
Dashboard
- Power BI Dashboard
dashboard/dashboard_real-time.pbix
Other components
- Local Machine
Connections 8
- Local Machine → Pipeline Orchestrator: provides metrics
README.md - Pipeline Orchestrator → Kafka Producer: calls send_message
Main/data_pipeline.py - Pipeline Orchestrator → Kafka Consumer: calls consum
Main/data_pipeline.py - Kafka Producer → Kafka Broker: publishes messages
Main/producer.py - Kafka Consumer → Kafka Broker: consumes messages
Main/consumer.py - Kafka Consumer → SQL Server: inserts metrics
Main/consumer.py - SQL Schema Setup → SQL Server: defines schema
README.md - Power BI Dashboard → SQL Server: reads metrics
README.md