# houdayfahousny/gcp_real_time_iot_data_pipeline_analytics architecture

> Architecture diagram of the GitHub repository houdayfahousny/gcp_real_time_iot_data_pipeline_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics
- Repository: https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics

## Overview

This repository implements an IoT streaming pipeline: a sample publisher sends machine readings through HiveMQ, a consumer forwards them to Kafka, Airflow monitors Kafka and submits a Spark job, and Spark cleans and aggregates streaming data for BigQuery. The graph also shows Spark’s console output. The README documents the overall architecture; component-level links are grounded in the excerpts, though deployment wiring in Docker Compose was not inspected.

Main entry points: [`airflow/dags/mqtt_data_pipeline.py`](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/airflow/dags/mqtt_data_pipeline.py) (Airflow DAG), [`spark/app/process_data.py`](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py) (Kafka stream read), [`scripts/mqtt_consumer.py`](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_consumer.py) (MQTT consumer), [`scripts/mqtt_publisher.py`](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_publisher.py) (MQTT publisher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["MQTT ingestion"]
  node_publisher["MQTT publisher<br/>[mqtt_publisher.py]"]
  node_broker["HiveMQ broker"]
  node_consumer["MQTT consumer<br/>[mqtt_consumer.py]"]
  node_kafka["Kafka"]
end

subgraph group_orchestration["Job orchestration"]
  node_dag["Airflow DAG"]
  node_monitor["Kafka monitor"]
end

subgraph group_processing["Stream processing"]
  node_spark_job["Spark job<br/>[process_data.py]"]
  node_kafka_read["Kafka stream read<br/>[process_data.py]"]
  node_parse_clean["Parse and clean<br/>[process_data.py]"]
  node_aggregate["Window aggregation<br/>[process_data.py]"]
end

subgraph group_outputs["Data outputs"]
  node_bigquery[("BigQuery")]
  node_console["Console output<br/>[process_data.py]"]
end

node_publisher -->|"publishes readings"| node_broker
node_consumer -->|"subscribes to"| node_broker
node_consumer -->|"forwards records"| node_kafka
node_dag -->|"defines task"| node_monitor
node_monitor -->|"inspects topics"| node_kafka
node_dag -->|"submits job"| node_spark_job
node_dag -->|"runs before"| node_monitor
node_spark_job -->|"starts stream"| node_kafka_read
node_kafka_read -->|"reads stream"| node_kafka
node_kafka_read -->|"parses records"| node_parse_clean
node_parse_clean -->|"aggregates windows"| node_aggregate
node_parse_clean -->|"writes records"| node_console
node_aggregate -->|"writes metrics"| node_bigquery

click node_publisher "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_publisher.py"
click node_consumer "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_consumer.py"
click node_dag "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/airflow/dags/mqtt_data_pipeline.py"
click node_monitor "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/airflow/dags/mqtt_data_pipeline.py"
click node_spark_job "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py"
click node_kafka_read "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py"
click node_parse_clean "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py"
click node_aggregate "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py"
click node_console "https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_publisher,node_broker,node_consumer,node_kafka toneBlue
class node_dag,node_monitor toneAmber
class node_spark_job,node_kafka_read,node_parse_clean,node_aggregate toneMint
class node_bigquery,node_console toneRose
```

## Components

### MQTT ingestion

- [MQTT publisher](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_publisher.py): `scripts/mqtt_publisher.py`
- **HiveMQ broker**
- [MQTT consumer](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/scripts/mqtt_consumer.py): `scripts/mqtt_consumer.py`
- **Kafka**

### Job orchestration

- [Airflow DAG](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/airflow/dags/mqtt_data_pipeline.py): `airflow/dags/mqtt_data_pipeline.py`
- [Kafka monitor](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/airflow/dags/mqtt_data_pipeline.py): `airflow/dags/mqtt_data_pipeline.py`

### Stream processing

- [Spark job](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py): `spark/app/process_data.py`
- [Kafka stream read](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py): `spark/app/process_data.py`
- [Parse and clean](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py): `spark/app/process_data.py`
- [Window aggregation](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py): `spark/app/process_data.py`

### Data outputs

- **BigQuery**
- [Console output](https://github.com/houdayfahousny/gcp_real_time_iot_data_pipeline_analytics/blob/main/spark/app/process_data.py): `spark/app/process_data.py`

## Connections

- MQTT publisher → HiveMQ broker: publishes readings (evidence: `scripts/mqtt_publisher.py`)
- MQTT consumer → HiveMQ broker: subscribes to (evidence: `scripts/mqtt_consumer.py`)
- MQTT consumer → Kafka: forwards records (evidence: `scripts/mqtt_consumer.py`)
- Airflow DAG → Kafka monitor: defines task (evidence: `airflow/dags/mqtt_data_pipeline.py`)
- Kafka monitor → Kafka: inspects topics (evidence: `airflow/dags/mqtt_data_pipeline.py`)
- Airflow DAG → Spark job: submits job (evidence: `airflow/dags/mqtt_data_pipeline.py`)
- Airflow DAG → Kafka monitor: runs before (evidence: `airflow/dags/mqtt_data_pipeline.py`)
- Spark job → Kafka stream read: starts stream (evidence: `spark/app/process_data.py`)
- Kafka stream read → Kafka: reads stream (evidence: `spark/app/process_data.py`)
- Kafka stream read → Parse and clean: parses records (evidence: `spark/app/process_data.py`)
- Parse and clean → Window aggregation: aggregates windows (evidence: `spark/app/process_data.py`)
- Parse and clean → Console output: writes records (evidence: `spark/app/process_data.py`)
- Window aggregation → BigQuery: writes metrics (evidence: `spark/app/process_data.py`)
