# ahmedennaifer/iot-streaming-platform architecture

> Architecture diagram of the GitHub repository ahmedennaifer/iot-streaming-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmedennaifer/iot-streaming-platform
- Repository: https://github.com/ahmedennaifer/iot-streaming-platform

## Overview

##### Main components
- Sensor Generator (Python):
– Under src/main/generator, you have modules for “producer” (with producer.py and run_both.sh) and “sensors” (with specific sensor types defined in enumerations and sensor modules). This part simulates or collects sensor data.
- Data Consumer (Scala):
– Located under src/main/scala/consumer, this contains configuration (SparkConfig.scala and config.scala), processing (MetricsProcessor.scala), and a main Spark consumer (sparkConsumer.scala). There is also a database interaction layer (database.scala and DatabaseWriter.scala) that likely persists processed metrics.
- Configuration & Resources:
– The application.conf file in resources and various build configuration files (build.sbt, pyproject.toml) define environment setup and project dependencies.
- Deployment & Containerization:
– Dockerfiles (both in docker/python and docker/scala) and docker-compose.yml highlight a multi-container deployment strategy.
- Monitoring:
– The README includes images of the “Platform Architecture” and the “Grafana Dashboard”, suggesting that monitoring, alerting, and dashboard visualization are part of the deployed system.

##### Relationships and interactions
- Data Flow Starts in the Python Sensor Generator:
– The sensor modules (e.g., sensors/enums/temperature.py, sensors/humidity.py) simulate or read sensor data.
– The producer (producer.py) aggregates or dispatches this sensor data. The run_both.sh script suggests a coordinated start or simultaneous running of sensor data generation.
- Data Hand-Off to the Scala Consumer:
– The consumer (sparkConsumer.scala) receives this data – potentially via a queue, file drop, or another transport layer (even if not immediately obvious from the structure, the separation suggests a producer/consumer paradigm).
– Once ingested, the MetricsProcessor processes the data and passes the results on.
- Persistent Storage and Visualization:
– The DatabaseWriter and database.scala components demonstrate that processed data is written to a database.
- Deployment and Orchestration:
– Docker files indicate containerization of both the Python producer and Scala consumer components.
– Docker-compose orchestrates these containers, defining how they interact in production.

##### Architectural patterns and design principles
- Separation of Concerns:
– The generator and consumer are clearly separated into different modules and possibly different containers.
- Modular and Extensible Design:
– The use of Scala (with Spark) for data processing creates a scalable cloud-based processing pipeline.
– The Python generator is likely designed to be easily extensible (additional sensor types can be added without major rewrites).
- Containerization:
– Having dedicated Docker setups for each component ensures a reproduceable deployment environment.
- Monitoring and Observability:
– The integration with Grafana demonstrates the emphasis on real-time monitoring of key metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Sensor Generator Subgraph
    subgraph "Sensor Generator (Python)"
      A1["Producer"]:::sensor
      A2["Run Script"]:::sensor
      A3["Sensors"]:::sensor
    end

    %% Data Consumer Subgraph
    subgraph "Data Consumer & Processing (Scala/Spark)"
      B1["Spark Consumer"]:::consumer
      B2["Metrics Processor"]:::consumer
      B3["Database Writer"]:::consumer
      B4["Database Module"]:::consumer
      B5["Consumer Config"]:::consumer
    end

    %% Persistence and Monitoring
    C["Database (Storage)"]:::database
    D["Grafana Dashboard"]:::monitoring

    %% Configuration & Resources
    subgraph "Configuration & Resources"
      E1["Application Config"]:::config
      E2["Build Config (build.sbt)"]:::config
      E3["Python Build Config (pyproject.toml)"]:::config
    end

    %% Deployment & Containerization
    subgraph "Container Infrastructure"
      F1["Python Dockerfile"]:::container
      F2["Scala Dockerfile"]:::container
      F3["Docker Compose"]:::container
    end

    %% Data Flow Arrows
    A1 -->|"sendsData"| B1
    A3 -->|"providesData"| B1
    B1 -->|"processes"| B2
    B2 -->|"writesData"| B3
    B2 -->|"updatesDB"| B4
    B3 -->|"stores"| C
    B4 -->|"stores"| C
    C -->|"feeds"| D

    %% Configuration influences both Sensor Generator and Data Consumer
    E1 --- A1
    E1 --- B5
    E2 --- B5
    E3 --- A1

    %% Container orchestration connecting to both Sensor and Consumer
    F1 --- A1
    F1 --- A2
    F1 --- A3
    F2 --- B1
    F2 --- B2
    F2 --- B3
    F3 --- F1
    F3 --- F2

    %% Click Events for Sensor Generator
    click A1 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/generator/producer/producer.py"
    click A2 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/generator/producer/run_both.sh"
    click A3 "https://github.com/ahmedennaifer/iot-streaming-platform/tree/main/src/main/generator/sensors/"

    %% Click Events for Data Consumer
    click B1 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/scala/consumer/sparkConsumer.scala"
    click B2 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/scala/consumer/processors/MetricsProcessor.scala"
    click B3 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/scala/consumer/writers/DatabaseWriter.scala"
    click B4 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/scala/consumer/database.scala"
    click B5 "https://github.com/ahmedennaifer/iot-streaming-platform/tree/main/src/main/scala/consumer/config"

    %% Click Events for Configuration & Resources
    click E1 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/src/main/resources/application.conf"
    click E2 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/build.sbt"
    click E3 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/pyproject.toml"

    %% Click Events for Deployment & Containerization
    click F1 "https://github.com/ahmedennaifer/iot-streaming-platform/tree/main/docker/python/Dockerfile"
    click F2 "https://github.com/ahmedennaifer/iot-streaming-platform/tree/main/docker/scala/Dockerfile"
    click F3 "https://github.com/ahmedennaifer/iot-streaming-platform/blob/main/docker-compose.yml"

classDef sensor fill:#a8d8ea,stroke:#000,stroke-width:2px;
classDef consumer fill:#ffb347,stroke:#000,stroke-width:2px;
classDef database fill:#b4e7a0,stroke:#000,stroke-width:2px;
classDef monitoring fill:#ffff99,stroke:#000,stroke-width:2px;
classDef config fill:#d291bc,stroke:#000,stroke-width:2px;
classDef container fill:#c0c0c0,stroke:#000,stroke-width:2px;
```
