# akramiot/iot-telemetry-data_aws_cloud_data_lake- architecture

> Architecture diagram of the GitHub repository akramiot/iot-telemetry-data_aws_cloud_data_lake-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-
- Repository: https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-

## Overview

##### Project type and purpose
- This is an end-to-end, serverless, AWS Cloud data ingestion pipeline that builds a Data Lake from IoT telemetry messages.
- MQTT-based clients (Cisco IoT gateways, Linux/Raspberry Pi) publish sensor data to AWS IoT Core; data then flows through IoT Analytics into S3 and is exposed via Glue/Athena/visualization tools.

##### File structure
- Top level:
- AWS_Cloud_IOT_Data_Lake/ – JSON definitions for IoT Analytics Channel, Pipeline, Datastore, Dataset, Crawler, Rule
- cloudformation.yaml – Infrastructure-as-Code to provision AWS IoT Analytics resources, IAM roles, S3 buckets
- basicPubSub.py, mqtt.sh – sample MQTT publisher/subscriber using AWS IoT Python SDK
- AWS_IOT_EI_Arch.py – Python script showcasing end-to-end ingestion (AWS SDK calls via boto3)
- requirements.txt – Python dependencies (AWS IoT SDK, boto3, etc.)
- device_specs.csv – reference spec for IoT devices
- AWS_Data_Lake_Output/ – Outputs, snapshots, documentation PDFs

##### README
- Describes sequence: Device → AWS IoT Core → IoT Analytics Channel → Pipeline → Data Store → Dataset → S3 Data Lake → Analytics/Visualization
- Points out optional use of AWS Glue Data Catalog or Lake Formation for integration

##### Main components
a. IoT Edge Devices
– Cisco IR1101/IR829/IR1800 series gateways or Raspberry Pi/Linux running basicPubSub.py
– Connect over MQTT to AWS IoT Core endpoint
b. AWS IoT Core
– Ingests MQTT messages; secured with X.509 certificates and policies
– IoT Rule (rule.json) forwards messages to IoT Analytics Channel
c. AWS IoT Analytics
– Channel: collects raw messages
– Pipeline: applies transformations (e.g., enrich, filter)
– Datastore: stores processed data
– Dataset: defines queries over the datastore and triggers data export
d. AWS S3 Data Lake
– Receives Dataset exports (raw, enriched data) in partitioned folders
e. AWS Glue Data Catalog / Crawler
– Scans S3 buckets (mycrawler.json) to create/update table definitions
f. Analytics & Visualization
– Athena/QuickSight or other BI tools to query Data Lake tables
g. Infrastructure as Code
– CloudFormation template (cloudformation.yaml) provisions all above
– JSON artifacts under AWS_Cloud_IOT_Data_Lake/ used by IaC or CLI

##### Relationships and data flow
1. Edge Devices → (MQTT) → AWS IoT Core
2. IoT Core Rule → IoT Analytics Channel
3. Channel → Pipeline → Datastore
4. Datastore → Dataset
5. Dataset → S3 Data Lake
6. S3 → Glue Crawler → Data Catalog
7. Data Catalog → Athena/QuickSight

##### Architectural patterns & design principles
- Event-driven, serverless pipeline
- Clear separation of concerns: Ingestion (IoT Core), Processing (IoT Analytics Pipeline), Storage (S3/Data Lake), Catalog (Glue), Consumption (BI)
- Infrastructure-as-Code for repeatability and version control
- Use of managed AWS services to minimize operational overhead

##### Technologies, frameworks, libraries
- AWS IoT Core, AWS IoT Analytics, AWS S3, AWS Glue, AWS Athena/QuickSight
- AWS IoT Device SDK for Python, boto3, AWS CLI
- CloudFormation
- MQTT protocol

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        PubSub["MQTT Publisher/Subscriber\nbasicPubSub.py / mqtt.sh"]:::ingestion
        EdgeScript["E2E Ingestion Script\nAWS_IOT_EI_Arch.py"]:::infrastructure
    end

    %% AWS IoT Core and Rule
    subgraph "AWS IoT Core" 
        IoTCore["AWS IoT Core\n(certificates & policies)"]:::ingestion
        Rule["IoT Rule\n(rule.json)"]:::ingestion
    end

    %% Processing Layer: IoT Analytics
    subgraph "IoT Analytics" 
        Channel["Analytics Channel"]:::processing
        Pipeline["Analytics Pipeline"]:::processing
        Datastore["Analytics Datastore"]:::processing
        Dataset["Analytics Dataset"]:::processing
    end

    %% Storage Layer
    subgraph "Data Lake"
        S3["S3 Data Lake"]:::storage
    end

    %% Catalog Layer
    subgraph "Data Catalog"
        GlueCrawler["Glue Crawler"]:::catalog
        Catalog["Glue Data Catalog"]:::catalog
    end

    %% Consumption Layer
    subgraph "Analytics & Visualization"
        Consumption["Athena / QuickSight"]:::consumption
    end

    %% Infrastructure as Code
    subgraph "Infrastructure as Code"
        CF["CloudFormation Template"]:::infrastructure
        Manifest["Deployment Manifest"]:::infrastructure
        Outputs["Outputs & Documentation"]:::infrastructure
        Specs["Device Specs Reference"]:::infrastructure
        Req["requirements.txt"]:::infrastructure
    end

    %% Data Flow
    PubSub -->|"MQTT"| IoTCore
    IoTCore -->| forwards via rule | Rule
    Rule -->|to Channel| Channel
    Channel -->|ingest| Pipeline
    Pipeline -->|store| Datastore
    Datastore -->|query| Dataset
    Dataset -->|export| S3
    S3 -->|scan| GlueCrawler
    GlueCrawler -->|update| Catalog
    Catalog -->|query| Consumption

    %% Supporting Flow
    CF -->|provisions| IoTCore
    CF -->|provisions| Channel
    CF -->|provisions| Pipeline
    CF -->|provisions| Datastore
    CF -->|provisions| Dataset
    CF -->|provisions| GlueCrawler
    Manifest -->|describes| CF

    %% Click Events
    click PubSub "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/basicPubSub.py"
    click PubSub "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/mqtt.sh"
    click EdgeScript "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_IOT_EI_Arch.py"
    click CF "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/cloudformation.yaml"
    click Channel "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/mychannel.json"
    click Pipeline "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/iot.json"
    click Datastore "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/mydatastore.json"
    click Dataset "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/mydataset.json"
    click Rule "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/rule.json"
    click GlueCrawler "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/mycrawler.json"
    click Manifest "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/AWS_Cloud_IOT_Data_lake/manifest.json"
    click Outputs "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/tree/main/AWS_Data_Lake_Output/"
    click Specs "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/device_specs.csv"
    click Req "https://github.com/akramiot/iot-telemetry-data_aws_cloud_data_lake-/blob/main/requirements.txt"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef catalog fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef consumption fill:#FFC0CB,stroke:#333,stroke-width:1px
    classDef infrastructure fill:#D3D3D3,stroke:#333,stroke-width:1px
```
