# chaurasiavikash/super_emitter_tracking architecture

> Architecture diagram of the GitHub repository chaurasiavikash/super_emitter_tracking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chaurasiavikash/super_emitter_tracking
- Repository: https://github.com/chaurasiavikash/super_emitter_tracking

## Overview

##### Project Type and Purpose
- This is a Python-based, full-stack data processing and monitoring system for detecting, tracking, and analyzing methane “super-emitters” from satellite (TROPOMI) data.
- It combines an ETL pipeline (data collection → preprocessing → detection → tracking), an alert/notification subsystem, and an interactive Streamlit dashboard for real-time monitoring and historical analysis.

##### File Structure Analysis
- src/
- data/           → ingestion (TROPOMI collector, meteorology, reference data) + validation/preprocessing
- detection/      → core detection algorithms (change detector, plume analysis, ensemble voting)
- tracking/       → temporal association and trend analysis (persistence filter, trend detector)
- alerts/         → alert manager, threshold monitoring, notification sender
- visualization/  → Streamlit dashboard components, map & time-series plotting, report generator
- analysis/       → statistical analyzers, uncertainty estimators, seasonal decomposers
- validation/     → cross-validation, ground truth comparison, performance metrics
- utils/          → shared helpers (file IO, geospatial, time, statistics)
- main.py         → CLI/orchestrator driving end-to-end pipeline
- config/          → YAML and Python config for thresholds, DB connection, logging
- deployment/      → Dockerfile, docker-compose, Kubernetes YAML, API server and Streamlit app entrypoints
- scripts/         → batch jobs (database setup, report generation, validation runs)
- notebooks/       → ad-hoc exploratory and case-study notebooks
- tests/           → pytest suite for each module

##### README Insights
- Processing modes: test, operational (real-time), historical
- External dependencies: Google Earth Engine, SQL database (emitter_database), email/webhook endpoints
- Key algorithms outlined: ensemble detection, DBSCAN clustering, Mann–Kendall trend tests
- System architecture summary: Data Collection → Preprocessing → Detection → Tracking → Alerts

- Data Ingestion Layer (src/data/*)
- Preprocessing & Validation (data_validator, preprocessor)
- Detection Engine (src/detection/* + analysis modules)
- Tracking Engine (src/tracking/*)
- Alerting Subsystem (src/alerts/*)
- Visualization Layer (src/visualization/* & Streamlit app)
- Storage Layer
- Configuration store (config/config.yaml)
- Time-series & detection records (emitter_database)
- Flat files/exports (CSV, GeoJSON, NetCDF)
- External Services
- Google Earth Engine (satellite data API)
- Notification endpoints (SMTP, webhooks)
- Orchestrator/API Server (src/main.py, deployment/api_server.py)

1. External Data Services → Data Ingestion
2. Ingestion → Validation → Preprocessing
3. Preprocessed data → Detection Engine
4. Detections → Tracking Engine → Trend & change analysis
5. Tracking outputs →
- Storage Layer
- Alerting Subsystem → Notification endpoints
- Visualization Dashboard → End user
6. Configuration store feeds thresholds & DB credentials to all modules
7. Orchestrator/API coordinates batch/real-time runs and serves dashboard via Streamlit

c. Architectural Patterns & Design Principles
- Layered Pipeline: clear separation of data, processing, and presentation
- Modularity: each algorithm lives in its own package (opens room for plugin-style extension)
- Config-driven: all tuning parameters in YAML; no hard-coding
- Microservice-style deployment: API server + dashboard + batch runner in separate containers
- Reproducibility: scripts, notebooks, and tests ensure consistency

d. Key Technologies & Frameworks
- Python 3.9+ with packages defined in requirements.txt
- Google Earth Engine Python API
- Pandas, NumPy, SciPy, scikit-learn (DBSCAN)
- Streamlit for dashboard
- SQL database (configured via emitter_database.yaml)
- Docker, Docker-Compose, Kubernetes

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Services
    subgraph "External Services"
        GEE["Google Earth Engine"]:::external
        NOTIF["Notification Endpoints (SMTP/Webhook)"]:::external
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        TC["TROPOMI Collector"]:::ingestion
        ML["Meteorology Loader"]:::ingestion
        RL["Reference Data Loader"]:::ingestion
    end

    %% Preprocessing & Validation
    subgraph "Preprocessing & Validation"
        DV["Data Validator"]:::ingestion
        PR["Preprocessor"]:::ingestion
    end

    %% Detection Engine
    subgraph "Detection Engine"
        CD["Change Detector"]:::compute
        PA["Plume Analyzer"]:::compute
        SE["Super-Emitter Detector"]:::compute
    end

    %% Tracking Engine
    subgraph "Tracking Engine"
        ET["Emitter Tracker"]:::compute
        PF["Persistence Filter"]:::compute
        TA["Temporal Analyzer"]:::compute
        TD["Trend Detector"]:::compute
    end

    %% Storage Layer
    subgraph "Storage Layer"
        DB["emitter_database (SQL)"]:::storage
        FL["Flat Files (CSV/GeoJSON/NetCDF)"]:::storage
        CFG["Configuration Store"]:::config
    end

    %% Alerting Subsystem
    subgraph "Alerting Subsystem"
        AM["Alert Manager"]:::alert
        TM["Threshold Monitor"]:::alert
        NS["Notification Sender"]:::alert
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        DASH["Streamlit Dashboard"]:::viz
        ADASH["Alert Dashboard"]:::viz
        MV["Map Visualizer"]:::viz
        TSP["Time-Series Plotter"]:::viz
        TP["Trend Plotter"]:::viz
        RG["Report Generator"]:::viz
    end

    %% Orchestrator/API
    subgraph "Orchestration & API"
        ORC["Orchestrator (main.py)"]:::config
        API["API Server"]:::config
    end

    %% Flows
    GEE --> TC
    GEE --> ML
    GEE --> RL

    TC --> DV
    ML --> DV
    RL --> DV

    DV --> PR

    PR --> CD
    PR --> PA
    PR --> SE

    CD --> ET
    PA --> ET
    SE --> ET

    ET --> PF
    PF --> TA
    TA --> TD

    TD --> DB
    TD --> FL

    TD --> AM
    AM --> TM
    TM --> NS
    NS --> NOTIF

    TD --> DASH

    CFG --> DV
    CFG --> PR
    CFG --> CD
    CFG --> PA
    CFG --> SE
    CFG --> ET
    CFG --> PF
    CFG --> TA
    CFG --> TD
    CFG --> AM
    CFG --> TM
    CFG --> NS
    CFG --> ORC
    CFG --> API

    ORC --> TC
    ORC --> DV
    ORC --> PR
    ORC --> CD
    ORC --> ET
    ORC --> AM
    ORC --> DASH
    ORC --> API

    API --> DASH

    %% Click Events
    click TC "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/data/tropomi_collector.py"
    click ML "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/data/meteorology_loader.py"
    click RL "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/data/reference_data_loader.py"
    click PR "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/data/preprocessor.py"
    click DV "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/data/data_validator.py"
    click CD "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/detection/change_detector.py"
    click PA "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/detection/plume_analyzer.py"
    click SE "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/detection/super_emitter_detector.py"
    click ET "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/tracking/emitter_tracker.py"
    click PF "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/tracking/persistence_filter.py"
    click TA "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/tracking/temporal_analyzer.py"
    click TD "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/tracking/trend_detector.py"
    click AM "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/alerts/alert_manager.py"
    click TM "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/alerts/threshold_monitor.py"
    click NS "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/alerts/notification_sender.py"
    click DASH "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/dashboard.py"
    click ADASH "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/alert_dashboard.py"
    click MV "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/map_visualizer.py"
    click TSP "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/time_series_plotter.py"
    click TP "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/trend_plotter.py"
    click RG "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/visualization/report_generator.py"
    click ORC "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/src/main.py"
    click API "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/deployment/api_server.py"
    click CFG "https://github.com/chaurasiavikash/super_emitter_tracking/blob/main/config/config.yaml"
    classDef ingestion fill:#e0f8e0,stroke:#0f0,stroke-width:1px
    classDef compute fill:#e0e8f8,stroke:#00f,stroke-width:1px
    classDef alert fill:#f8e0e0,stroke:#f00,stroke-width:1px
    classDef viz fill:#f0e0f8,stroke:#c0f,stroke-width:1px
    classDef external fill:#fff0e0,stroke:#fa0,stroke-width:1px
    classDef storage fill:#f0f0f0,stroke:#888,stroke-width:1px
    classDef config fill:#d0d0d0,stroke:#666,stroke-width:1px
```
