How amiruthavallii/climate-monitor works
Project Type and Purpose
• This is a full-stack, serverless data pipeline and analytics dashboard.
• Purpose: Collect historical, current, and forecast climate data (weather, air quality, floods) via various AWS Lambda ETLs, store in a Postgres RDS, and present it through a Streamlit dashboard hosted on AWS Fargate. It also has scheduled notification and daily-summary services via AWS SES.
File Structure Analysis
• Top-level “terraform/” defines all AWS infrastructure (RDS, Lambda, EventBridge, Fargate, SES, ECR repos).
• “db/” contains the database schema, seed scripts, and location/flood-area assignment handlers for initial setup.
• ETL Lambdas:
• extract-past/… (weather & air_quality)
• extract-present/… (current weather, air_quality orchestrator)
• extract-future/… (future predictions)
• live-flood-monitoring-etl/… (live flood warnings)
• load-historic-flood-data/… (bulk historic floods)
• orchestrator-new-location/… (onboard new location, triggering all static ETLs)
• “notifications/” and “daily-summary/” each define a scheduled Lambda for alerts and daily email summaries.
• “dashboard/” holds the Streamlit front-end (homepage, nav module, pages for weather, air_quality, floods, login, profile).
Key Insights from README
• RDS schema is 3NF; ETLs run on scheduled EventBridge triggers or upon location onboarding.
• Dashboard is containerized and runs on Fargate.
• All Lambdas are built via Docker, pushed to ECR, and referenced in Terraform.
• SES for sending notification and summary emails.
Main Components
a. External Users
• End users access the Streamlit dashboard via a URL.
• Users receive emails (notifications, daily summaries) via SES.
b. Frontend Service
• Streamlit app running in AWS Fargate/ECS.
• Connects to Postgres RDS for queries.
c. Data Store
• AWS RDS Postgres instance (schema.sql + seed.sql → tables for weather, air quality, floods, locations, assignments, users).
d. ETL & Ingestion Layer (Serverless Lambdas)
• extract-past Lambdas → historical_weather_readings, historical_air_quality
• extract-present Lambdas → weather_readings, air_quality_readings; orchestrator invokes both
• extract-future Lambda → future_weather_predictions
• live-flood-monitoring-etl Lambda → flood_warnings
• load-historic-flood-data → historical_floods
• orchestrator-new-location Lambda → triggers static data ETLs upon new location
e. Scheduling & Orchestration
• AWS EventBridge rules schedule:
• periodic ETL runs (present, future, live flood)
• daily summary Lambda
• notification Lambda
f. Notification System
• notifications Lambda queries RDS for threshold breaches → Amazon SES → user emails
• daily-summary Lambda queries last-24h data → SES → user emails
g. Infrastructure as Code
• Terraform controls creation of RDS, Lambda functions (with ECR images), EventBridge rules, ECS/Fargate service, SES permissions, IAM roles.
Relationships & Data Flow
• External APIs (OpenMeteo, OpenWeather, Gov flood APIs) → Lambda ETLs → RDS tables.
• EventBridge → scheduled Lambdas → data ingestion.
• User → HTTPS → Fargate/Streamlit → queries RDS → UI dashboards.
• EventBridge → notification Lambdas → SES → user email inbox.
• Terraform orchestrates all AWS resources.
Architectural Patterns & Principles
• Serverless microservices (single-purpose Lambdas)
• Event-driven architecture (EventBridge triggers)
• Containerization (Docker + ECR for Lambdas & Fargate service)
• Clear separation of concerns: ingestion, data storage, presentation, notifications.
Technologies/Frameworks/Libraries to Call Out
• AWS Lambda, AWS EventBridge, AWS SES, AWS RDS (Postgres), AWS Fargate/ECS, Amazon ECR
• Terraform for IaC
• Python for all code (Streamlit front end, ETL, orchestrators, notifications)
• Docker for packaging