# shahilcseai/gdg-hackathon architecture

> Architecture diagram of the GitHub repository shahilcseai/gdg-hackathon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shahilcseai/gdg-hackathon
- Repository: https://github.com/shahilcseai/gdg-hackathon

## Overview

##### Project Type and Purpose
- This is a Python-based data-processing and ML pipeline whose goal is to downscale coarse-resolution satellite NO₂ data to a fine-resolution air-quality map.
- It ingests multiple data sources (satellite swath/gridded TIFF, ground station CSV), applies pre‐processing, trains or applies an ML model, validates results, and produces final maps.

##### File Structure Analysis
- Top-level scripts:
– main.py: orchestrates end-to-end workflow (ingest → preprocess → train/evaluate → output)
– database.py: handles reading/writing intermediate data (e.g., local cache or PostGIS)
– model.py: defines/selects/trains the ML algorithm (RandomForest, XGBoost, CNN, etc.)
– utils.py: shared utilities (e.g., data cleaning, gap filling for cloudy pixels, coordinate transforms)
- styles.css: styling for any HTML/JS map visualizations or reports
- pyproject.toml: project metadata and dependencies
- README.md: high-level description, data sources, algorithm choices, usage

A. External Data Sources
- Satellite NO₂ sources (TROPOMI/Sentinel-5p swath or gridded; OMI Aura)
- Ground-station CPCB CSV/API
B. Ingestion Layer
- Scripts in database.py and utils.py that fetch, cache, and standardize raw data
C. Pre-processing Layer
- Gap‐filling logic for cloudy pixels
- Spatial re-projection and gridding
- Merging satellite + ground data
D. ML Model Layer
- model.py: training pipeline or inference pipeline
- Hyperparameter tuning (RF, XGBoost, ANN/CNN)
E. Validation Layer
- Hold-out data evaluation (unseen independent)
- Accuracy metrics (RMSE, spatial error maps)
F. Output/Visualization Layer
- Fine-resolution NO₂ maps (GeoTIFF, shapefile)
- HTML dashboard or report styled via styles.css
G. Orchestration
- main.py coordinates:
– calls ingestion → pre‐processing → model training/inference → validation → reporting

##### Relationships and Data Flows
1. External Data Sources → Ingestion Layer
2. Ingestion → Pre-processing
3. Pre-processing → ML Model Layer
4. ML Model Layer ↔ Validation Layer (bidirectional during training/tuning)
5. ML Model Layer → Output/Visualization
6. main.py (orchestrator) invokes each component in sequence
- That database.py and utils.py are shared libraries supporting multiple stages.

##### Architectural Patterns and Design Principles
- Modular pipeline: separate ingestion, preprocessing, modeling, validation, and reporting
- Separation of concerns:
- database.py handles I/O, utils.py handles generic transformations, model.py focuses on ML logic
- Configurable ML back-end: plug-and-play algorithms (RF, XGBoost, CNN)
- Stateless steps: each stage reads inputs, writes outputs—can parallelize or run on different machines

##### Technologies, Frameworks, and External Services
- Python ML libraries: scikit-learn, XGBoost, TensorFlow/PyTorch
- Geospatial libraries: rasterio, geopandas, shapely
- Data ingestion: requests, earthengine-api (for GEE)
- Reporting: folium or plotly for interactive maps, HTML/CSS (styles.css)
- (Optional) Database: SQLite/PostGIS via SQLAlchemy or custom code in database.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration
    main["main.py\n(Orchestrator)"]:::orchestration
    click main "https://github.com/shahilcseai/gdg-hackathon/blob/main/main.py"

    %% External Systems
    subgraph "External Systems"
        direction TB
        sat["Satellite NO₂ Data\n(TROPOMI/Sentinel-5p, OMI Aura)"]:::external
        cpcb["Ground-Station Data\n(CPCB CSV/API)"]:::external
    end

    %% Ingestion & Storage
    subgraph "Ingestion & Storage"
        direction TB
        db["database.py\n(Data I/O / Caching)"]:::processing
        utils_ingest["utils.py\n(Shared Utilities)"]:::processing
        click db "https://github.com/shahilcseai/gdg-hackathon/blob/main/database.py"
        click utils_ingest "https://github.com/shahilcseai/gdg-hackathon/blob/main/utils.py"
    end

    %% Data Processing
    subgraph "Data Processing"
        direction TB
        preprocess["Pre-processing\n(Gap-filling, Reprojection, Merging)"]:::processing
        utils_pre["utils.py\n(Pre-processing routines)"]:::processing
        click utils_pre "https://github.com/shahilcseai/gdg-hackathon/blob/main/utils.py"
    end

    %% ML / AI Layer
    subgraph "ML / AI"
        direction TB
        model_train["model.py\n(Training / Inference)"]:::ml
        validate["Validation\n(RMSE, Spatial Errors)"]:::ml
        click model_train "https://github.com/shahilcseai/gdg-hackathon/blob/main/model.py"
    end

    %% Visualization & Reporting
    subgraph "Visualization & Reporting"
        direction TB
        output_maps["Output Maps\n(GeoTIFF, Shapefile)"]:::viz
        report["HTML Dashboard / Report"]:::viz
        styles["styles.css\n(Styling)"]:::viz
        click styles "https://github.com/shahilcseai/gdg-hackathon/blob/main/styles.css"
    end

    %% Configuration & Documentation
    subgraph "Project Configuration"
        direction TB
        config["pyproject.toml"]:::config
        readme["README.md"]:::config
        click config "https://github.com/shahilcseai/gdg-hackathon/blob/main/pyproject.toml"
        click readme "https://github.com/shahilcseai/gdg-hackathon/blob/main/README.md"
    end

    %% Data Flows
    sat -->|ingest| db
    cpcb -->|ingest| utils_ingest
    db -->|raw data| preprocess
    utils_ingest -->|standardize| preprocess
    preprocess -->|processed data| model_train
    model_train -->|predictions| validate
    validate -->|feedback| model_train
    model_train -->|results| output_maps
    model_train -->|results| report

    %% Orchestration Control
    main -.->|invoke ingest| db
    main -.->|invoke preprocess| preprocess
    main -.->|invoke model| model_train
    main -.->|invoke validation| validate
    main -.->|invoke reporting| report

    %% Classes
    classDef external fill:#add8e6,stroke:#333,stroke-width:1px
    classDef processing fill:#90ee90,stroke:#333,stroke-width:1px
    classDef ml fill:#ffa500,stroke:#333,stroke-width:1px
    classDef viz fill:#dda0dd,stroke:#333,stroke-width:1px
    classDef orchestration fill:#ffffe0,stroke:#333,stroke-width:1px
    classDef config fill:#f0f8ff,stroke:#333,stroke-width:1px
```
