# asdaraujo/edge2ai-workshop architecture

> Architecture diagram of the GitHub repository asdaraujo/edge2ai-workshop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asdaraujo/edge2ai-workshop
- Repository: https://github.com/asdaraujo/edge2ai-workshop

## Overview

##### Project type and purpose
- This is an open-source workshop toolkit for building a hands-on IoT predictive-maintenance pipeline on a single-host “one-node” CDH/CDP cluster.
- It combines Infrastructure-as-Code (Terraform), data-simulation code, stream-processing jobs (Flink, SQL Stream Builder), NiFi flows (edge and datacenter), analytic workloads (Spark, Kudu), visualization (Cloudera Data Viz), and experiment/model management (CDSW).

##### File structure
- Top level: documentation (.adoc), simulator code (cdsw_*.py), schemas (.avsc), NiFi flow definitions (.xml), workshop guides, and a “flink” Maven project for streaming.
- “setup/terraform”: full IaC module for AWS/Azure cluster provisioning, including:
- providers/aws and providers/azure submodules
- shared resources (scripts, Python utilities, lab assets)
- “web” sub-folder: a Flask-based registration/service UI
- tests for resource modules
- “flink”: streaming job code targeting Kafka for CDC/workshop_cdc, with pom.xml and resources.
- Data folder: a historical data file for simulator input.

##### README for additional insights
- Lists all services running on the single host: Cloudera Manager, Atlas, CDSW, Data Viz, EFM, Flink, Hue, Knox, NiFi, NiFi Registry, Ranger, Schema Registry, SMM, SSB.
- Describes a workflow from sensor ingestion through streaming, storage, analytics, and visualization.

a. Main components (group by logical layer)
- Edge Layer
- Sensor Simulator (cdsw.iot_exp.py / data/historical_iot.txt)
- MiNiFi Agents (NiFi-flow XML for edge)
- Ingestion & Stream-Processing Layer
- NiFi on cluster (Run_Python_Simulator_NiFi.xml, IoT_Data_Simulator_Version_1.xml)
- Kafka cluster (provisioned via Terraform)
- Schema Registry (sensor.avsc, sensor_enhanced.avsc)
- Stream Compute Layer
- Flink Streaming Job (flink/src/main/scala/…/StreamingJobIoT.scala)
- SQL Stream Builder (SSB)
- Storage & Batch Analytics Layer
- Kudu (used by Spark jobs)
- HDFS (for archival)
- Spark analytics workloads (spark.iot.py)
- Visualization & Modeling Layer
- Cloudera Data Viz dashboards (workshop_dataviz.adoc + artifacts)
- CDSW experiments/models (cdsw.iot_model.py)
- Control & Management Layer
- Terraform IaC modules (setup/terraform)
- Flask registration Web UI (setup/terraform/web/app)
- Lab orchestration scripts (setup/terraform/resources/*.py, .sh)
- Cloudera Manager & EFM (Edge Flow Manager)

b. Relationships and interactions
1. Sensor Simulator → MiNiFi Agents (HTTP/MQTT)
2. MiNiFi → NiFi (NiFi site-to-site)
3. NiFi → Kafka (Publish to topics)
4. Kafka → Flink & SSB (Consume streams)
5. Flink → Kudu/HDFS (Write processed data)
6. Spark jobs on Kudu → Data Viz dashboards & CDSW for modeling
7. Schema Registry enforces Avro schemas across producers/consumers
8. Terraform modules → Provision all cluster services, networking, security groups
9. Flask web UI → Registration and cluster metadata stored in Postgres (or local file)

c. Architectural patterns and principles
- Layered/sliding-window architecture: edge, ingestion, compute, storage, viz
- Infrastructure as Code: Terraform modules, reusable providers, automated tests
- Micro-services style for registration UI (Flask app) and utility scripts
- Event-driven stream processing (NiFi → Kafka → Flink/SSB)
- Separation of concerns: ingestion vs. processing vs. analytic vs. orchestration

d. Key technologies/frameworks/libraries
- Terraform (AWS/Azure providers, modules, tests)
- Python (simulator, Terraform resource scripts, Flask web UI)
- Apache NiFi & MiNiFi (XML flow definitions)
- Apache Kafka & Schema Registry
- Apache Flink (Scala streaming job)
- Cloudera SSB (SQL on streams)
- Apache Kudu & Spark
- Cloudera Data Visualization & CDSW

##### Key takeaways
- Both the data path (sensor → MiNiFi → NiFi → Kafka → compute → storage → viz) and the control path (Terraform → cluster services → registration UI).
- Be detailed: include schemas, scripts, modules, and templates that underpin each component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client/Edge" 
        direction TB
        SensorSim["Sensor Simulator\n(cdsy.iot_exp.py)"]:::edge
        MiNiFI1["MiNiFi: IoT Data Simulator Flow"]:::edge
        MiNiFI2["MiNiFi: Run Python Simulator"]:::edge
    end

    subgraph "Cluster/Platform"
        direction TB
        NiFi["Apache NiFi"]:::ingest
        Kafka["Kafka Cluster"]:::ingest
        SchemaReg["Schema Registry"]:::ingest
        FlinkJob["Flink Streaming Job\n(StreamingJobIoT.scala)"]:::compute
        SSB["SQL Stream Builder"]:::compute
        Kudu[(Kudu)]:::storage
        HDFS[(HDFS)]:::storage
        SparkJob["Spark Analytics\n(spark.iot.py)"]:::storage
        DataViz["Cloudera Data Viz"]:::viz
        CDSW["CDSW Experiments\n(cdsw.iot_model.py)"]:::viz
    end

    subgraph "Control Plane"
        direction TB
        Terraform["Terraform IaC\n(setup/terraform/)"]:::control
        FlaskUI["Flask Web UI\n(app/)"]:::control
        CM["Cloudera Manager & EFM"]:::control
    end

    SensorSim -->|HTTP/MQTT| MiNiFI1
    SensorSim -->|HTTP/MQTT| MiNiFI2
    MiNiFI1 -->|"site-to-site"| NiFi
    MiNiFI2 -->|"site-to-site"| NiFi
    NiFi -->|"publish topics"| Kafka
    Kafka -->|"consume"| FlinkJob
    Kafka -->|"consume"| SSB
    FlinkJob -->|"write"| Kudu
    FlinkJob -->|"archive"| HDFS
    SparkJob -->|"read/write"| Kudu
    SparkJob -->|"output"| DataViz
    SparkJob -->|"output"| CDSW
    SchemaReg -.-> Kafka
    SchemaReg -.-> FlinkJob
    SchemaReg -.-> SSB
    Terraform -.-> CM
    Terraform --> NiFi
    Terraform --> Kafka
    Terraform --> SchemaReg
    Terraform --> FlinkJob
    Terraform --> Kudu
    Terraform --> HDFS
    Terraform --> DataViz
    Terraform --> CDSW
    FlaskUI -->|API calls| Terraform
    CM -->|manages| NiFi
    CM -->|manages| Kafka
    CM -->|manages| SchemaReg
    CM -->|manages| FlinkJob
    CM -->|manages| Kudu
    CM -->|manages| HDFS
    CM -->|manages| DataViz
    CM -->|manages| CDSW

    click SensorSim "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/cdsy.iot_exp.py"
    click MiNiFI1 "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/IoT_Data_Simulator_Version_1.xml"
    click MiNiFI2 "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/Run_Python_Simulator_NiFi.xml"
    click Kafka "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/setup/terraform/resources/templates/kafka.json"
    click SchemaReg "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/sensor.avsc"
    click FlinkJob "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/flink/src/main/scala/flink-workshop/StreamingJobIoT.scala"
    click SSB "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/workshop_ssb.adoc"
    click Kudu "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/setup/terraform/resources/templates/kudu.json"
    click HDFS "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/setup/terraform/resources/templates/hdfs.json"
    click SparkJob "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/setup/terraform/resources/spark.iot.py"
    click DataViz "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/workshop_dataviz.adoc"
    click CDSW "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/cdsy.iot_model.py"
    click Terraform "https://github.com/asdaraujo/edge2ai-workshop/tree/trunk/setup/terraform/"
    click FlaskUI "https://github.com/asdaraujo/edge2ai-workshop/tree/trunk/setup/terraform/web/app/"
    click CM "https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/setup/terraform/resources/templates/cm_template.py"

    classDef edge fill:#e0f7fa,stroke:#0288d1,color:#000
    classDef ingest fill:#bbdefb,stroke:#1976d2,color:#000
    classDef compute fill:#c8e6c9,stroke:#388e3c,color:#000
    classDef storage fill:#ffe0b2,stroke:#f57c00,color:#000
    classDef viz fill:#d1c4e9,stroke:#7e57c2,color:#000
    classDef control fill:#f8bbd0,stroke:#ad1457,color:#000
```
