How fpgmaas/stream-iot works
Project Type and Purpose
• This is an end-to-end IoT streaming data pipeline running in Azure, designed to mock sensor data, transport it through Kafka, orchestrate processing with Airflow, and persist results to MongoDB/CosmosDB.
• Key use case: learn and demonstrate Kafka-based event streaming, containerized microservices, infrastructure as code (Terraform), and observability (Prometheus/Grafana).
File Structure Analysis
• Top-level directories:
• terraform: Azure infra modules (AKS, Blob, Container Registry, CosmosDB) and orchestration of the cluster and services.
• kafka: Helm/Strimzi values files for deploying Kafka on Kubernetes.
• airflow: Kubernetes manifests (PersistentVolumeClaims, values) and DAG definitions.
• python: Dockerfile and Python app for sensor data producer and consumer logic.
• monitoring: Helm/values for Prometheus and Grafana.
• CI/CD: GitHub workflows for Terraform provisioning and Docker image builds/pushes.
• Config files: helm values.yaml for each service, pyproject.toml for Python dependencies, .env.config for Terraform variables.
README Insights
• Technologies: Azure, Terraform, Kubernetes (AKS), Docker, Airflow, Kafka, Python, Prometheus/Grafana, CosmosDB/MongoDB.
• Architecture image reference: confirms the pipeline flow—sensor → Kafka → Airflow tasks → DB storage → monitoring.