How asdaraujo/edge2ai-workshop works
Project type and purpose
• This is an open-source workshop toolkit for building a hands-on IoT predictive-maintenance pipeline on a single-host “one-node” CDH/CDP cluster.
• It combines Infrastructure-as-Code (Terraform), data-simulation code, stream-processing jobs (Flink, SQL Stream Builder), NiFi flows (edge and datacenter), analytic workloads (Spark, Kudu), visualization (Cloudera Data Viz), and experiment/model management (CDSW).
File structure
• Top level: documentation (.adoc), simulator code (cdsw_*.py), schemas (.avsc), NiFi flow definitions (.xml), workshop guides, and a “flink” Maven project for streaming.
• “setup/terraform”: full IaC module for AWS/Azure cluster provisioning, including:
• providers/aws and providers/azure submodules
• shared resources (scripts, Python utilities, lab assets)
• “web” sub-folder: a Flask-based registration/service UI
• tests for resource modules
• “flink”: streaming job code targeting Kafka for CDC/workshop_cdc, with pom.xml and resources.
• Data folder: a historical data file for simulator input.
README for additional insights
• Lists all services running on the single host: Cloudera Manager, Atlas, CDSW, Data Viz, EFM, Flink, Hue, Knox, NiFi, NiFi Registry, Ranger, Schema Registry, SMM, SSB.
• Describes a workflow from sensor ingestion through streaming, storage, analytics, and visualization.
a. Main components (group by logical layer)
• Edge Layer
• Sensor Simulator (cdsw.iot_exp.py / data/historical_iot.txt)
• MiNiFi Agents (NiFi-flow XML for edge)
• Ingestion & Stream-Processing Layer
• NiFi on cluster (Run_Python_Simulator_NiFi.xml, IoT_Data_Simulator_Version_1.xml)
• Kafka cluster (provisioned via Terraform)
• Schema Registry (sensor.avsc, sensor_enhanced.avsc)
• Stream Compute Layer
• Flink Streaming Job (flink/src/main/scala/…/StreamingJobIoT.scala)
• SQL Stream Builder (SSB)
• Storage & Batch Analytics Layer
• Kudu (used by Spark jobs)
• HDFS (for archival)
• Spark analytics workloads (spark.iot.py)
• Visualization & Modeling Layer
• Cloudera Data Viz dashboards (workshop_dataviz.adoc + artifacts)
• CDSW experiments/models (cdsw.iot_model.py)
• Control & Management Layer
• Terraform IaC modules (setup/terraform)
• Flask registration Web UI (setup/terraform/web/app)
• Lab orchestration scripts (setup/terraform/resources/*.py, .sh)
• Cloudera Manager & EFM (Edge Flow Manager)
b. Relationships and interactions
1. Sensor Simulator → MiNiFi Agents (HTTP/MQTT)
2. MiNiFi → NiFi (NiFi site-to-site)
3. NiFi → Kafka (Publish to topics)
4. Kafka → Flink & SSB (Consume streams)
5. Flink → Kudu/HDFS (Write processed data)
6. Spark jobs on Kudu → Data Viz dashboards & CDSW for modeling
7. Schema Registry enforces Avro schemas across producers/consumers
8. Terraform modules → Provision all cluster services, networking, security groups
9. Flask web UI → Registration and cluster metadata stored in Postgres (or local file)
c. Architectural patterns and principles
• Layered/sliding-window architecture: edge, ingestion, compute, storage, viz
• Infrastructure as Code: Terraform modules, reusable providers, automated tests
• Micro-services style for registration UI (Flask app) and utility scripts
• Event-driven stream processing (NiFi → Kafka → Flink/SSB)
• Separation of concerns: ingestion vs. processing vs. analytic vs. orchestration
d. Key technologies/frameworks/libraries
• Terraform (AWS/Azure providers, modules, tests)
• Python (simulator, Terraform resource scripts, Flask web UI)
• Apache NiFi & MiNiFi (XML flow definitions)
• Apache Kafka & Schema Registry
• Apache Flink (Scala streaming job)
• Cloudera SSB (SQL on streams)
• Apache Kudu & Spark
• Cloudera Data Visualization & CDSW
Key takeaways
• Both the data path (sensor → MiNiFi → NiFi → Kafka → compute → storage → viz) and the control path (Terraform → cluster services → registration UI).
• Be detailed: include schemas, scripts, modules, and templates that underpin each component.