# aymane-maghouti/real-time-data-pipeline-using-kafka architecture

> Architecture diagram of the GitHub repository aymane-maghouti/real-time-data-pipeline-using-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aymane-maghouti/real-time-data-pipeline-using-kafka
- Repository: https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka

## Overview

This repository monitors local system performance through a threaded Python pipeline: it collects metrics, publishes them to Kafka, consumes and stores them in SQL Server, and presents them in a Power BI dashboard. The principal runtime stages are implemented in `Main/data_pipeline.py`, `Main/producer.py`, and `Main/consumer.py`. The README documents SQL Server setup and dashboard connectivity; the dashboard’s internal implementation is not shown, so its details are not expanded here.

Main entry points: [`Main/data_pipeline.py`](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/data_pipeline.py) (Pipeline Orchestrator), [`Main/consumer.py`](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/consumer.py) (Kafka Consumer), [`Main/producer.py`](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/producer.py) (Kafka Producer), [`Main/sql-scripts.sql`](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/sql-scripts.sql) (SQL Schema Setup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Metric Runtime"]
  node_pipeline["Pipeline Orchestrator<br/>[data_pipeline.py]"]
end

subgraph group_streaming["Kafka Streaming"]
  node_producer["Kafka Producer<br/>[producer.py]"]
  node_kafka["Kafka Broker"]
  node_consumer["Kafka Consumer<br/>[consumer.py]"]
end

subgraph group_storage["SQL Storage"]
  node_sql_setup["SQL Schema Setup<br/>[sql-scripts.sql]"]
  node_sql_server[("SQL Server")]
end

subgraph group_visualization["Dashboard"]
  node_powerbi["Power BI Dashboard"]
end

node_local_machine(("Local Machine"))

node_local_machine -->|"provides metrics"| node_pipeline
node_pipeline -->|"calls send_message"| node_producer
node_pipeline -->|"calls consum"| node_consumer
node_producer -->|"publishes messages"| node_kafka
node_consumer -->|"consumes messages"| node_kafka
node_consumer -->|"inserts metrics"| node_sql_server
node_sql_setup -->|"defines schema"| node_sql_server
node_powerbi -->|"reads metrics"| node_sql_server

click node_pipeline "https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/data_pipeline.py"
click node_producer "https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/producer.py"
click node_consumer "https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/consumer.py"
click node_sql_setup "https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/sql-scripts.sql"
click node_powerbi "https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/dashboard/dashboard_real-time.pbix"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline toneBlue
class node_producer,node_kafka,node_consumer toneAmber
class node_sql_setup,node_sql_server toneMint
class node_powerbi toneRose
class node_local_machine toneIndigo
```

## Components

### Metric Runtime

- [Pipeline Orchestrator](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/data_pipeline.py): `Main/data_pipeline.py`

### Kafka Streaming

- [Kafka Producer](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/producer.py): `Main/producer.py`
- **Kafka Broker**
- [Kafka Consumer](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/consumer.py): `Main/consumer.py`

### SQL Storage

- [SQL Schema Setup](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/Main/sql-scripts.sql): `Main/sql-scripts.sql`
- **SQL Server**

### Dashboard

- [Power BI Dashboard](https://github.com/aymane-maghouti/real-time-data-pipeline-using-kafka/blob/main/dashboard/dashboard_real-time.pbix): `dashboard/dashboard_real-time.pbix`

### Other components

- **Local Machine**

## Connections

- Local Machine → Pipeline Orchestrator: provides metrics (evidence: `README.md`)
- Pipeline Orchestrator → Kafka Producer: calls send_message (evidence: `Main/data_pipeline.py`)
- Pipeline Orchestrator → Kafka Consumer: calls consum (evidence: `Main/data_pipeline.py`)
- Kafka Producer → Kafka Broker: publishes messages (evidence: `Main/producer.py`)
- Kafka Consumer → Kafka Broker: consumes messages (evidence: `Main/consumer.py`)
- Kafka Consumer → SQL Server: inserts metrics (evidence: `Main/consumer.py`)
- SQL Schema Setup → SQL Server: defines schema (evidence: `README.md`)
- Power BI Dashboard → SQL Server: reads metrics (evidence: `README.md`)
