How akramiot/iot-telemetry-data_aws_cloud_data_lake- works
Project type and purpose
• This is an end-to-end, serverless, AWS Cloud data ingestion pipeline that builds a Data Lake from IoT telemetry messages.
• MQTT-based clients (Cisco IoT gateways, Linux/Raspberry Pi) publish sensor data to AWS IoT Core; data then flows through IoT Analytics into S3 and is exposed via Glue/Athena/visualization tools.
File structure
• Top level:
• AWS_Cloud_IOT_Data_Lake/ – JSON definitions for IoT Analytics Channel, Pipeline, Datastore, Dataset, Crawler, Rule
• cloudformation.yaml – Infrastructure-as-Code to provision AWS IoT Analytics resources, IAM roles, S3 buckets
• basicPubSub.py, mqtt.sh – sample MQTT publisher/subscriber using AWS IoT Python SDK
• AWS_IOT_EI_Arch.py – Python script showcasing end-to-end ingestion (AWS SDK calls via boto3)
• requirements.txt – Python dependencies (AWS IoT SDK, boto3, etc.)
• device_specs.csv – reference spec for IoT devices
• AWS_Data_Lake_Output/ – Outputs, snapshots, documentation PDFs
README
• Describes sequence: Device → AWS IoT Core → IoT Analytics Channel → Pipeline → Data Store → Dataset → S3 Data Lake → Analytics/Visualization
• Points out optional use of AWS Glue Data Catalog or Lake Formation for integration
Main components
a. IoT Edge Devices
– Cisco IR1101/IR829/IR1800 series gateways or Raspberry Pi/Linux running basicPubSub.py
– Connect over MQTT to AWS IoT Core endpoint
b. AWS IoT Core
– Ingests MQTT messages; secured with X.509 certificates and policies
– IoT Rule (rule.json) forwards messages to IoT Analytics Channel
c. AWS IoT Analytics
– Channel: collects raw messages
– Pipeline: applies transformations (e.g., enrich, filter)
– Datastore: stores processed data
– Dataset: defines queries over the datastore and triggers data export
d. AWS S3 Data Lake
– Receives Dataset exports (raw, enriched data) in partitioned folders
e. AWS Glue Data Catalog / Crawler
– Scans S3 buckets (mycrawler.json) to create/update table definitions
f. Analytics & Visualization
– Athena/QuickSight or other BI tools to query Data Lake tables
g. Infrastructure as Code
– CloudFormation template (cloudformation.yaml) provisions all above
– JSON artifacts under AWS_Cloud_IOT_Data_Lake/ used by IaC or CLI
Relationships and data flow
1. Edge Devices → (MQTT) → AWS IoT Core
2. IoT Core Rule → IoT Analytics Channel
3. Channel → Pipeline → Datastore
4. Datastore → Dataset
5. Dataset → S3 Data Lake
6. S3 → Glue Crawler → Data Catalog
7. Data Catalog → Athena/QuickSight
Architectural patterns & design principles
• Event-driven, serverless pipeline
• Clear separation of concerns: Ingestion (IoT Core), Processing (IoT Analytics Pipeline), Storage (S3/Data Lake), Catalog (Glue), Consumption (BI)
• Infrastructure-as-Code for repeatability and version control
• Use of managed AWS services to minimize operational overhead
Technologies, frameworks, libraries
• AWS IoT Core, AWS IoT Analytics, AWS S3, AWS Glue, AWS Athena/QuickSight
• AWS IoT Device SDK for Python, boto3, AWS CLI
• CloudFormation
• MQTT protocol