# amiruthavallii/climate-monitor architecture

> Architecture diagram of the GitHub repository amiruthavallii/climate-monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amiruthavallii/climate-monitor
- Repository: https://github.com/amiruthavallii/climate-monitor

## Overview

##### Project Type and Purpose
- This is a full-stack, serverless data pipeline and analytics dashboard.
- Purpose: Collect historical, current, and forecast climate data (weather, air quality, floods) via various AWS Lambda ETLs, store in a Postgres RDS, and present it through a Streamlit dashboard hosted on AWS Fargate. It also has scheduled notification and daily-summary services via AWS SES.

##### File Structure Analysis
- Top-level “terraform/” defines all AWS infrastructure (RDS, Lambda, EventBridge, Fargate, SES, ECR repos).
- “db/” contains the database schema, seed scripts, and location/flood-area assignment handlers for initial setup.
- ETL Lambdas:
- extract-past/… (weather & air_quality)
- extract-present/… (current weather, air_quality orchestrator)
- extract-future/… (future predictions)
- live-flood-monitoring-etl/… (live flood warnings)
- load-historic-flood-data/… (bulk historic floods)
- orchestrator-new-location/… (onboard new location, triggering all static ETLs)
- “notifications/” and “daily-summary/” each define a scheduled Lambda for alerts and daily email summaries.
- “dashboard/” holds the Streamlit front-end (homepage, nav module, pages for weather, air_quality, floods, login, profile).

##### Key Insights from README
- RDS schema is 3NF; ETLs run on scheduled EventBridge triggers or upon location onboarding.
- Dashboard is containerized and runs on Fargate.
- All Lambdas are built via Docker, pushed to ECR, and referenced in Terraform.
- SES for sending notification and summary emails.

##### Main Components
a. External Users
- End users access the Streamlit dashboard via a URL.
- Users receive emails (notifications, daily summaries) via SES.
b. Frontend Service
- Streamlit app running in AWS Fargate/ECS.
- Connects to Postgres RDS for queries.
c. Data Store
- AWS RDS Postgres instance (schema.sql + seed.sql → tables for weather, air quality, floods, locations, assignments, users).
d. ETL & Ingestion Layer (Serverless Lambdas)
- extract-past Lambdas → historical_weather_readings, historical_air_quality
- extract-present Lambdas → weather_readings, air_quality_readings; orchestrator invokes both
- extract-future Lambda → future_weather_predictions
- live-flood-monitoring-etl Lambda → flood_warnings
- load-historic-flood-data → historical_floods
- orchestrator-new-location Lambda → triggers static data ETLs upon new location
e. Scheduling & Orchestration
- AWS EventBridge rules schedule:
- periodic ETL runs (present, future, live flood)
- daily summary Lambda
- notification Lambda
f. Notification System
- notifications Lambda queries RDS for threshold breaches → Amazon SES → user emails
- daily-summary Lambda queries last-24h data → SES → user emails
g. Infrastructure as Code
- Terraform controls creation of RDS, Lambda functions (with ECR images), EventBridge rules, ECS/Fargate service, SES permissions, IAM roles.

##### Relationships & Data Flow
- External APIs (OpenMeteo, OpenWeather, Gov flood APIs) → Lambda ETLs → RDS tables.
- EventBridge → scheduled Lambdas → data ingestion.
- User → HTTPS → Fargate/Streamlit → queries RDS → UI dashboards.
- EventBridge → notification Lambdas → SES → user email inbox.
- Terraform orchestrates all AWS resources.

##### Architectural Patterns & Principles
- Serverless microservices (single-purpose Lambdas)
- Event-driven architecture (EventBridge triggers)
- Containerization (Docker + ECR for Lambdas & Fargate service)
- Clear separation of concerns: ingestion, data storage, presentation, notifications.

##### Technologies/Frameworks/Libraries to Call Out
- AWS Lambda, AWS EventBridge, AWS SES, AWS RDS (Postgres), AWS Fargate/ECS, Amazon ECR
- Terraform for IaC
- Python for all code (Streamlit front end, ETL, orchestrators, notifications)
- Docker for packaging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "External APIs" 
        direction TB
        A1["OpenMeteo API"]:::external
        A2["OpenWeather API"]:::external
        A3["Gov Flood API"]:::external
    end

    %% EventBridge
    EB["AWS EventBridge"]:::eventbridge

    %% ETL Lambdas
    subgraph "ETL Lambdas" 
        direction TB
        P1["extract_past_weather Lambda"]:::lambda
        P2["extract_past_air_quality Lambda"]:::lambda
        C1["extract_present_weather Lambda"]:::lambda
        C2["extract_present_air_quality Lambda"]:::lambda
        F1["extract_future_predictions Lambda"]:::lambda
        L1["live_flood_monitoring Lambda"]:::lambda
        H1["load_historic_floods Lambda"]:::lambda
    end

    %% Orchestrators
    subgraph "Orchestrators" 
        direction TB
        O1["current_data_orchestrator Lambda"]:::orchestrator
        O2["new_location_onboarding Lambda"]:::orchestrator
    end

    %% Data Store
    DB["Postgres RDS"]:::db

    %% Notifications
    subgraph "Notification Services" 
        direction TB
        N1["threshold_alerts Lambda"]:::notification
        N2["daily_summary Lambda"]:::notification
        SES["Amazon SES"]:::ses
    end

    %% Dashboard
    subgraph "Streamlit Dashboard (Fargate)" 
        direction TB
        D1["homepage.py"]:::dashboard
        D2["nav module"]:::dashboard
        D3["weather page"]:::dashboard
        D4["air_quality page"]:::dashboard
        D5["floods page"]:::dashboard
        D6["login page"]:::dashboard
        D7["profile page"]:::dashboard
    end

    %% IaC
    TF["Terraform IaC"]:::terraform

    %% External Users
    User["End Users"]:::external_user

    %% Connections: APIs to ETL
    A1 --> P1
    A3 --> L1
    A2 --> P2

    %% EventBridge to ETLs & Orchestrators & Notifications
    EB --> P1
    EB --> P2
    EB --> C1
    EB --> C2
    EB --> F1
    EB --> L1
    EB --> H1
    EB --> N1
    EB --> N2
    EB --> O2

    %% Orchestrator flows
    C1 --> O1
    C2 --> O1
    O1 --> C1
    EB --> O1

    O2 --> P1
    O2 --> P2
    O2 --> H1

    %% ETLs to RDS
    P1 --> DB
    P2 --> DB
    C1 --> DB
    C2 --> DB
    F1 --> DB
    L1 --> DB
    H1 --> DB
    O2 --> DB

    %% Dashboard flow
    User -->|HTTPS| D1
    D1 --> DB

    %% Notifications flow
    N1 --> SES --> User
    N2 --> SES --> User
    N1 --> DB
    N2 --> DB

    %% Terraform manages all AWS resources
    TF --> EB
    TF --> P1
    TF --> P2
    TF --> C1
    TF --> C2
    TF --> F1
    TF --> L1
    TF --> H1
    TF --> O1
    TF --> O2
    TF --> N1
    TF --> N2
    TF --> SES
    TF --> DB
    TF --> D1

    %% Click Events for component mappings
    click TF "https://github.com/amiruthavallii/climate-monitor/blob/main/terraform/main.tf"
    click P1 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-past/weather/extract.py"
    click P2 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-past/air_quality/extract_air_quality.py"
    click C1 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-present/extract.py"
    click O1 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-present/orchestrator/orchestrator_lambda.py"
    click C2 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-present-air-quality/extract.py"
    click F1 "https://github.com/amiruthavallii/climate-monitor/blob/main/extract-future/extract_future.py"
    click L1 "https://github.com/amiruthavallii/climate-monitor/blob/main/live-flood-monitoring-etl/fetch_live_flood_warnings.py"
    click H1 "https://github.com/amiruthavallii/climate-monitor/blob/main/load-historic-flood-data/upload_historic_floods.py"
    click O2 "https://github.com/amiruthavallii/climate-monitor/blob/main/orchestrator-new-location/new_location_orchestrator.py"
    click N1 "https://github.com/amiruthavallii/climate-monitor/blob/main/notifications/notification.py"
    click N2 "https://github.com/amiruthavallii/climate-monitor/blob/main/daily-summary/summary.py"
    click D1 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/homepage.py"
    click D2 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/modules/nav.py"
    click D3 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/pages/weather.py"
    click D4 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/pages/air_quality.py"
    click D5 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/pages/floods.py"
    click D6 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/pages/login.py"
    click D7 "https://github.com/amiruthavallii/climate-monitor/blob/main/dashboard/pages/profile.py"

    %% Styles
    classDef lambda fill:#FFE599,stroke:#D6B656,stroke-width:2px;
    classDef orchestrator fill:#F4CCCC,stroke:#C29B9B,stroke-width:2px;
    classDef notification fill:#D9EAD3,stroke:#6AA84F,stroke-width:2px;
    classDef dashboard fill:#C9DAF8,stroke:#6D9EEB,stroke-width:2px;
    classDef eventbridge fill:#D0E0E3,stroke:#76A5AF,stroke-width:2px;
    classDef db fill:#B4A7D6,stroke:#674EA7,stroke-width:2px;
    classDef ses fill:#D9D2E9,stroke:#8E7CC3,stroke-width:2px;
    classDef terraform fill:#F9CB9C,stroke:#B45F06,stroke-width:2px;
    classDef external fill:#FFFFFF,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5;
    classDef external_user fill:#FFFFFF,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5;
```
