How houdayfahousny/gcp_real_time_iot_data_pipeline_analytics works
This repository implements an IoT streaming pipeline: a sample publisher sends machine readings through HiveMQ, a consumer forwards them to Kafka, Airflow monitors Kafka and submits a Spark job, and Spark cleans and aggregates streaming data for BigQuery. The graph also shows Spark’s console output. The README documents the overall architecture; component-level links are grounded in the excerpts, though deployment wiring in Docker Compose was not inspected.
Main entry points: mqtt_data_pipeline.py Airflow DAG · process_data.py Kafka stream read · mqtt_consumer.py MQTT consumer · mqtt_publisher.py MQTT publisher
Components 12
MQTT ingestion
- MQTT publisher
scripts/mqtt_publisher.py - HiveMQ broker
- MQTT consumer
scripts/mqtt_consumer.py - Kafka
Job orchestration
- Airflow DAG
airflow/dags/mqtt_data_pipeline.py - Kafka monitor
airflow/dags/mqtt_data_pipeline.py
Stream processing
- Spark job
spark/app/process_data.py - Kafka stream read
spark/app/process_data.py - Parse and clean
spark/app/process_data.py - Window aggregation
spark/app/process_data.py
Data outputs
- BigQuery
- Console output
spark/app/process_data.py
Connections 13
- MQTT publisher → HiveMQ broker: publishes readings
scripts/mqtt_publisher.py - MQTT consumer → HiveMQ broker: subscribes to
scripts/mqtt_consumer.py - MQTT consumer → Kafka: forwards records
scripts/mqtt_consumer.py - Airflow DAG → Kafka monitor: defines task
airflow/dags/mqtt_data_pipeline.py - Kafka monitor → Kafka: inspects topics
airflow/dags/mqtt_data_pipeline.py - Airflow DAG → Spark job: submits job
airflow/dags/mqtt_data_pipeline.py - Airflow DAG → Kafka monitor: runs before
airflow/dags/mqtt_data_pipeline.py - Spark job → Kafka stream read: starts stream
spark/app/process_data.py - Kafka stream read → Kafka: reads stream
spark/app/process_data.py - Kafka stream read → Parse and clean: parses records
spark/app/process_data.py - Parse and clean → Window aggregation: aggregates windows
spark/app/process_data.py - Parse and clean → Console output: writes records
spark/app/process_data.py - Window aggregation → BigQuery: writes metrics
spark/app/process_data.py