# fpgmaas/stream-iot architecture

> Architecture diagram of the GitHub repository fpgmaas/stream-iot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fpgmaas/stream-iot
- Repository: https://github.com/fpgmaas/stream-iot

## Overview

##### Project Type and Purpose
- This is an end-to-end IoT streaming data pipeline running in Azure, designed to mock sensor data, transport it through Kafka, orchestrate processing with Airflow, and persist results to MongoDB/CosmosDB.
- Key use case: learn and demonstrate Kafka-based event streaming, containerized microservices, infrastructure as code (Terraform), and observability (Prometheus/Grafana).

##### File Structure Analysis
- Top-level directories:
- terraform: Azure infra modules (AKS, Blob, Container Registry, CosmosDB) and orchestration of the cluster and services.
- kafka: Helm/Strimzi values files for deploying Kafka on Kubernetes.
- airflow: Kubernetes manifests (PersistentVolumeClaims, values) and DAG definitions.
- python: Dockerfile and Python app for sensor data producer and consumer logic.
- monitoring: Helm/values for Prometheus and Grafana.
- CI/CD: GitHub workflows for Terraform provisioning and Docker image builds/pushes.
- Config files: helm values.yaml for each service, pyproject.toml for Python dependencies, .env.config for Terraform variables.

##### README Insights
- Technologies: Azure, Terraform, Kubernetes (AKS), Docker, Airflow, Kafka, Python, Prometheus/Grafana, CosmosDB/MongoDB.
- Architecture image reference: confirms the pipeline flow—sensor → Kafka → Airflow tasks → DB storage → monitoring.

## Diagram (Mermaid)

```mermaid
graph TD
    %% CI/CD Layer
    subgraph "CI/CD Pipeline" 
        direction TB
        CI["GitHub Actions"]:::ci
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer (Terraform)" 
        direction TB
        TF_ROOT["Terraform Root"]:::infra
        TF_AKS["AKS Module"]:::infra
        TF_ACR["Container Registry Module"]:::infra
        TF_BLOB["Blob Storage Module"]:::infra
        TF_COSMOS["CosmosDB Module"]:::infra
    end

    %% Azure Resources
    subgraph "Azure Resources" 
        direction TB
        RG["Azure Resource Group"]:::cloud
        AKS_CLUSTER["AKS Cluster"]:::cloud
        ACR["Azure Container Registry"]:::cloud
        BLOB["Azure Blob Storage"]:::cloud
        COSMOS["CosmosDB (MongoAPI)"]:::data
    end

    %% Kubernetes Cluster
    subgraph "AKS Cluster" 
        direction TB
        subgraph "kafka namespace" 
            direction TB
            KAFKA_CLUSTER["Strimzi Kafka Cluster"]:::k8s
        end
        subgraph "airflow namespace" 
            direction TB
            AIRFLOW_SCHED["Airflow Scheduler"]:::k8s
            AIRFLOW_WS["Airflow Webserver"]:::k8s
            AIRFLOW_WORKERS["Airflow Workers"]:::k8s
        end
        subgraph "python-app namespace" 
            direction TB
            PRODUCER_POD["Producer Pod"]:::app
            CONSUMER_POD["Consumer Pod"]:::app
        end
        subgraph "monitoring namespace" 
            direction TB
            PROM["Prometheus"]:::monitor
            GRAF["Grafana"]:::monitor
        end
    end

    %% Data Pipeline Flow
    subgraph "Data Pipeline" 
        direction TB
        SENSOR["Sensor Mock (Producer Trigger)"]:::app
    end

    %% Connections
    CI -->|"triggers Terraform"| TF_ROOT
    TF_ROOT --> TF_AKS
    TF_ROOT --> TF_ACR
    TF_ROOT --> TF_BLOB
    TF_ROOT --> TF_COSMOS

    TF_AKS -->|"provisions"| RG
    RG --> AKS_CLUSTER
    TF_ACR --> ACR
    TF_BLOB --> BLOB
    TF_COSMOS --> COSMOS

    SENSOR -->|"publishes to topic"| KAFKA_CLUSTER
    KAFKA_CLUSTER -->|"subscribed by"| CONSUMER_POD
    CONSUMER_POD -->|"writes to"| COSMOS

    AIRFLOW_SCHED -->|"orchestrates"| PRODUCER_POD
    AIRFLOW_SCHED -->|"orchestrates"| CONSUMER_POD

    PROM -->|"scrapes metrics from"| KAFKA_CLUSTER
    PROM -->|"scrapes metrics from"| AIRFLOW_SCHED
    PROM -->|"scrapes metrics from"| PRODUCER_POD
    PROM -->|"scrapes metrics from"| CONSUMER_POD
    GRAF -->|"queries metrics from"| PROM

    %% Click Events
    click TF_ROOT "https://github.com/fpgmaas/stream-iot/blob/main/terraform/main.tf"
    click TF_ROOT "https://github.com/fpgmaas/stream-iot/blob/main/terraform/providers.tf"
    click TF_ROOT "https://github.com/fpgmaas/stream-iot/blob/main/terraform/variables.tf"
    click TF_ROOT "https://github.com/fpgmaas/stream-iot/blob/main/terraform/main.auto.tfvars"
    click TF_ROOT "https://github.com/fpgmaas/stream-iot/blob/main/terraform/.env.config"
    click TF_AKS "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/aks/main.tf"
    click TF_AKS "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/aks/variables.tf"
    click TF_ACR "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/container-registry/main.tf"
    click TF_ACR "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/container-registry/variables.tf"
    click TF_BLOB "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/blob/main.tf"
    click TF_BLOB "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/blob/variables.tf"
    click TF_COSMOS "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/cosmosdb/main.tf"
    click TF_COSMOS "https://github.com/fpgmaas/stream-iot/blob/main/terraform/modules/cosmosdb/variables.tf"
    click KAFKA_CLUSTER "https://github.com/fpgmaas/stream-iot/blob/main/kafka/values-strimzi.yaml"
    click KAFKA_CLUSTER "https://github.com/fpgmaas/stream-iot/blob/main/kafka/values-kafka.yaml"
    click AIRFLOW_SCHED "https://github.com/fpgmaas/stream-iot/blob/main/airflow/values.yaml"
    click AIRFLOW_SCHED "https://github.com/fpgmaas/stream-iot/blob/main/airflow/pv-logs.yaml"
    click AIRFLOW_SCHED "https://github.com/fpgmaas/stream-iot/blob/main/airflow/pvc-logs.yaml"
    click AIRFLOW_SCHED "https://github.com/fpgmaas/stream-iot/blob/main/dags/producer.py"
    click AIRFLOW_SCHED "https://github.com/fpgmaas/stream-iot/blob/main/dags/consumer.py"
    click PRODUCER_POD "https://github.com/fpgmaas/stream-iot/tree/main/python/Dockerfile"
    click PRODUCER_POD "https://github.com/fpgmaas/stream-iot/blob/main/python/pyproject.toml"
    click PRODUCER_POD "https://github.com/fpgmaas/stream-iot/blob/main/python/app/producer.py"
    click PRODUCER_POD "https://github.com/fpgmaas/stream-iot/blob/main/python/app/config.py"
    click CONSUMER_POD "https://github.com/fpgmaas/stream-iot/blob/main/python/app/consumer.py"
    click PROM "https://github.com/fpgmaas/stream-iot/blob/main/monitoring/values.yaml"
    click PROM "https://github.com/fpgmaas/stream-iot/blob/main/monitoring/README.md"
    click CI "https://github.com/fpgmaas/stream-iot/blob/main/.github/workflows/deploy-terraform.yml"
    click CI "https://github.com/fpgmaas/stream-iot/blob/main/.github/workflows/deploy-terraform-state-bucket.yml"
    click CI "https://github.com/fpgmaas/stream-iot/blob/main/.github/workflows/on-pull-request-terraform.yml"
    click CI "https://github.com/fpgmaas/stream-iot/blob/main/.github/workflows/push-docker-image.yml"

    %% Styles
    classDef infra fill:#FFE4B5,stroke:#E69A2A,color:#000;
    classDef cloud fill:#D0EFFF,stroke:#1E90FF,color:#000;
    classDef k8s fill:#E6E6FA,stroke:#9370DB,color:#000;
    classDef app fill:#F0FFF0,stroke:#3CB371,color:#000;
    classDef data fill:#FFFACD,stroke:#FFD700,color:#000;
    classDef monitor fill:#FFF0F5,stroke:#FF69B4,color:#000;
    classDef ci fill:#F5F5F5,stroke:#A9A9A9,color:#000;
```
