# wat-ai/satml architecture

> Architecture diagram of the GitHub repository wat-ai/satml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wat-ai/satml
- Repository: https://github.com/wat-ai/satml

## Overview

##### Project Type & Purpose
- SatML is a machine-learning pipeline for detecting methane plumes in hyperspectral satellite images.
- It encompasses two primary workflows: model training (offline pipeline) and model inference (online or batch prediction).

##### File Structure
- Top-level directories:
- config: global settings, constants, model hyperparameters (`bbox_model_config.yaml`, `constants.py`)
- data: raw hyperspectral TIFFs, CSV manifest, gitignored large files
- models: model definitions (e.g. `UNetModule.py`), training entry points (`train_model.py`, `train_segmentation.py`), model factory
- src: reusable library code (data loader, utilities, loss functions, pipeline orchestration)
- scripts: one-off tools (visualizer, benchmark)
- notebooks: exploratory analysis and prototyping
- logs: runtime logs, checkpoints
- Notice a clear separation between “core library” (src), “model logic” (models), and “orchestration tools” (scripts/notebooks).

##### Extract Insights from README
- Two PNGs show “Model Training Flow” and “Model Inference Flow.”
- The stack is Python 3.11, with ML frameworks implicitly referenced in requirements (e.g. PyTorch or TensorFlow, NumPy, rasterio).

##### Main Components & Their Relationships
a. Data Ingestion & Storage
- Raw satellite images (TIFFs) and labels stored under `data/raw_data` and referenced via `train_easy.csv`.
- Config files feed parameters into data-loader components.
b. Preprocessing & Utilities
- In `src/data_loader.py`, `src/data_utils.py`, `src/image_utils.py` you standardize, normalize, stack spectral bands.
- File utilities (`src/file_utils.py`) handle I/O.
c. Model Definition & Training
- Model classes in `models/UNetModule.py` enable segmentation
- `models/cnn_model_factory.py` chooses model type
- Training scripts (`train_model.py`, `train_segmentation.py`) orchestrate epoch loops, loss computation (from `src/losses.py`), metrics, checkpointing to `logs/`.
d. Inference Pipeline
- `src/segment_pipeline.py` loads trained weights, applies segmentation to new images, writes outputs.
- Visualization or quality-check via `scripts/bbox_visualizer.py`.
e. External Artifacts & Documentation
- Google Slides plan and research links
- Notebooks for data exploration (`notebooks/`)

##### Architectural Patterns & Design Principles
- Modular layering: config → src (core library) → models → scripts
- Factory pattern for model instantiation
- Separation of concerns: training vs. inference vs. experimentation
- Pipeline orchestration (data → preprocess → train/test → evaluate → save artifact)

##### Final Touches
- Version info (Python 3.11) and major dependencies (e.g. PyTorch, NumPy, rasterio) in a corner.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Training Workflow
    subgraph "SatML Training Architecture"
        direction TB
        configYAML["bbox_model_config.yaml"]:::config
        configConst["constants.py"]:::config

        rawData["Raw Hyperspectral Images"]:::dataSource
        csvManifest["CSV Manifest"]:::dataSource

        dataLoader["Data Loader"]:::compute
        dataUtils["Data Utilities"]:::compute
        imageUtils["Image Utilities"]:::compute
        fileUtils["File I/O Utilities"]:::compute

        unetModule["U-Net Segmentation Module"]:::compute
        bboxWrapper["Bounding-Box Model Wrapper"]:::compute
        cnnFactory["CNN Model Factory"]:::compute

        trainModel["train_model.py"]:::compute
        trainSeg["train_segmentation.py"]:::compute

        losses["Loss Functions & Metrics"]:::compute

        checkpoints["Model Checkpoints"]:::storage

        %% Connections Training
        configYAML -->|"hyperparameters"| dataLoader
        configConst -->|"constants"| dataLoader

        rawData -->|"TIFF images"| dataLoader
        csvManifest -->|"labels CSV"| dataLoader

        dataLoader -->|"tensors"| dataUtils
        dataUtils -->|"normalized tensors"| imageUtils
        imageUtils -->|"stacked bands"| fileUtils

        fileUtils -->|"preprocessed data"| trainModel
        fileUtils -->|"preprocessed data"| trainSeg

        trainModel -->|"select model"| cnnFactory
        trainSeg -->|"select model"| cnnFactory

        cnnFactory --> unetModule
        cnnFactory --> bboxWrapper

        unetModule -->|"predictions"| losses
        bboxWrapper -->|"predictions"| losses

        losses -->|"loss values"| trainSeg
        trainSeg -->|"trained weights"| checkpoints
        trainModel -->|"trained weights"| checkpoints
    end

    %% Inference Workflow
    subgraph "SatML Inference Architecture"
        direction TB
        configYAML2["bbox_model_config.yaml"]:::config
        configConst2["constants.py"]:::config

        newImages["New Hyperspectral Images"]:::dataSource
        segmentPipeline["segment_pipeline.py"]:::compute
        visTool["bbox_visualizer.py"]:::compute
        outputMasks["Segmentation Masks"]:::storage

        %% Connections Inference
        configYAML2 -->|"hyperparameters"| segmentPipeline
        configConst2 -->|"constants"| segmentPipeline

        newImages -->|"TIFF images"| segmentPipeline
        segmentPipeline -->|"segmentation mask"| outputMasks
        outputMasks -->|"masks"| visTool
    end

    %% External Artifacts & Documentation
    subgraph "Exploration & Docs"
        direction TB
        dataNB["data_exploration.ipynb"]:::compute
        plumeNB["plume_segment.ipynb"]:::compute
        trainDoc["training.png"]:::storage
        inferDoc["inference.png"]:::storage
    end

    %% Legend
    subgraph Legend
        direction TB
        D1["Data Source"]:::dataSource
        C1["Compute Module"]:::compute
        S1["Storage/Checkpoints"]:::storage
        CFG["Configuration"]:::config
    end

    %% Node positioning
    rawData --- csvManifest

    %% Click Events
    click configYAML "https://github.com/wat-ai/satml/blob/main/config/bbox_model_config.yaml"
    click configConst "https://github.com/wat-ai/satml/blob/main/config/constants.py"
    click rawData "https://github.com/wat-ai/satml/tree/main/data/raw_data/STARCOP_train_easy/"
    click csvManifest "https://github.com/wat-ai/satml/blob/main/data/raw_data/STARCOP_train_easy/train_easy.csv"
    click dataLoader "https://github.com/wat-ai/satml/blob/main/src/data_loader.py"
    click dataUtils "https://github.com/wat-ai/satml/blob/main/src/data_utils.py"
    click imageUtils "https://github.com/wat-ai/satml/blob/main/src/image_utils.py"
    click fileUtils "https://github.com/wat-ai/satml/blob/main/src/file_utils.py"
    click losses "https://github.com/wat-ai/satml/blob/main/src/losses.py"
    click unetModule "https://github.com/wat-ai/satml/blob/main/models/UNetModule.py"
    click bboxWrapper "https://github.com/wat-ai/satml/blob/main/models/bbox_model.py"
    click cnnFactory "https://github.com/wat-ai/satml/blob/main/models/cnn_model_factory.py"
    click trainModel "https://github.com/wat-ai/satml/blob/main/models/train_model.py"
    click trainSeg "https://github.com/wat-ai/satml/blob/main/models/train_segmentation.py"
    click segmentPipeline "https://github.com/wat-ai/satml/blob/main/src/segment_pipeline.py"
    click visTool "https://github.com/wat-ai/satml/blob/main/scripts/bbox_visualizer.py"
    click dataNB "https://github.com/wat-ai/satml/blob/main/notebooks/data_exploration.ipynb"
    click plumeNB "https://github.com/wat-ai/satml/blob/main/notebooks/plume_segment.ipynb"
    click trainDoc "https://github.com/wat-ai/satml/blob/main/docs/training.png"
    click inferDoc "https://github.com/wat-ai/satml/blob/main/docs/inference.png"

    %% Styles
    classDef dataSource fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#98FB98,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
```
