# nivivirat/final-year-project architecture

> Architecture diagram of the GitHub repository nivivirat/final-year-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nivivirat/final-year-project
- Repository: https://github.com/nivivirat/final-year-project

## Overview

##### Main Components
a. Data Sources and Storage:
- Input data is stored in multiple HDF5 (.h5) files. There are directories like “data”, “Dataset_Analysis”, and “indian_data” that hold different datasets (e.g., SEVIR, Indian data).
- These datasets serve as the raw inputs for training and testing.
b. Preprocessing:
- Under the “ef-sat2rad/Preprocess” directory, scripts (such as save_oneByOne.py and ef_sat2rad.txt) indicate that data normalization, transformation, or augmentation is performed before training.
c. Core Model Components:
- The “ef-sat2rad/earthformer” directory contains critical modules:
- “cuboid_transformer”: likely handling spatial-temporal transformations.
- “datasets”, “metrics”, and “visualization”: responsible for loading data, computing performance metrics (such as enso, skill scores, etc.), and visualizing outcomes.
- The “ef-sat2rad/model_uNet” directory includes a U-Net variant (UNet3D_baseline.py) that might be used for segmentation or spatial prediction tasks.
d. Training and Experimentation:
- The script “train_cuboid_sevir_invLinear.py” acts as the training entry point.
- The “experiments” directory manages configuration files, logging outputs (with several “lightning_logs” versions), and saved intermediate outputs.
- The “trained_ckpt” folder stores the trained model checkpoints.
e. Auxiliary Components:
- Utility modules in both “ef-sat2rad/utils” and various “earthformer/utils” help with optimization, checkpoint handling, and other supportive functions.

##### Relationships and Data Flows
- Data → Model Pipeline: From the data loaders, the data is input to the core models – the cuboid transformer (and associated baselines) as well as the U-Net model; indicate data flow to these components.
- Model Predictions → Visualization: Finally, outputs progress to “visualization” modules (across “ef-sat2rad/earthformer/visualization” and separate visualizer scripts) that help interpret the results.

##### Architectural Patterns and Design Principles
- Modular Design: The separation between preprocessing, model definition (earthformer and model_uNet), training, and evaluation/logging is key.
- Pipeline-Oriented Workflow: Data flows sequentially from raw/preprocessed inputs, through data augmentation, model training, and finally evaluation/visualization.
- Configurability: The presence of YAML configuration files (like “ef-sat2rad/cfgs/cfg.yaml” and “experiments/cfg.yaml”) suggests that parameters and model settings are externalized, emphasizing flexibility.
- Scalability and Reproducibility: Experiment logs and checkpointing indicate a focus on reproducible experiments.

##### Relevant Technologies/Frameworks
- Deep Learning Framework: Likely PyTorch (or a derivative) is being used, as indicated by “lightning_logs” (possibly via PyTorch Lightning) and modules like “apex_ddp.py” for distributed training.
- Data Formats: HDF5 files are used extensively.
- Configuration & Environment: The use of Conda (environment.yml and conda commands) points to dependency management best practices.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion
    subgraph "Data Ingestion"
        data1["data/"]
        data2["Dataset_Analysis/"]
        data3["indian_data/"]
    end

    %% Preprocessing
    subgraph "Preprocessing"
        pre1["save_oneByOne.py"]
        pre2["ef_sat2rad.txt"]
    end

    %% Data Loader & Augmentation
    subgraph "Data Loader & Augmentation"
        loader["Datasets Directory"]
        sevir["SEVIR Data Loader"]
        augment["Augmentation"]
    end

    %% Core Models
    subgraph "Core Models"
        cuboid["Cuboid Transformer"]
        unet["U-Net Variant"]
    end

    %% Training & Experimentation
    subgraph "Training & Experimentation"
        train["Training Script"]
        experiments["Experiments (Config & Logs)"]
        checkpoint["Model Checkpoints"]
    end

    %% Metrics & Visualization
    subgraph "Metrics & Visualization"
        metrics["Evaluation Metrics"]
        vizModules["Visualization Modules"]
        viz1["Visualize Script"]
        viz2["Visualize Output"]
    end

    %% Auxiliary Utilities
    subgraph "Auxiliary Utilities"
        utils["Utility Functions"]
        earthUtils["Earthformer Utilities"]
    end

    %% Data Flow Connections
    data1 -->|"ingests"| pre1
    data2 -->|"ingests"| pre1
    data3 -->|"ingests"| pre1
    pre1 -->|"processes"| pre2
    pre2 -->|"outputs to"| loader
    loader -->|"feeds"| sevir
    loader -->|"supports"| augment
    augment -->|"enhances"| sevir
    sevir -->|"supplies"| cuboid
    sevir -->|"supplies"| unet
    cuboid -->|"trains via"| train
    unet -->|"trains via"| train
    train -->|"logs to"| experiments
    train -->|"saves"| checkpoint
    train -->|"evaluates"| metrics
    experiments -->|"reports"| metrics
    metrics -->|"channels to"| vizModules
    vizModules -->|"runs"| viz1
    vizModules -->|"runs"| viz2

    %% Auxiliary Utilities dotted connections
    utils -.->|"supports"| pre1
    utils -.->|"supports"| train
    earthUtils -.->|"optimizes"| cuboid
    earthUtils -.->|"optimizes"| unet

    %% Class Definitions for color styling
    class data1,data2,data3 dataClass
    class pre1,pre2 preprocessClass
    class loader,sevir,augment loaderClass
    class cuboid,unet modelClass
    class train,experiments,checkpoint trainingClass
    class metrics,vizModules,viz1,viz2 vizClass
    class utils,earthUtils utilClass

    classDef dataClass fill:#AEDFF7,stroke:#333,stroke-width:2px;
    classDef preprocessClass fill:#C1F0C1,stroke:#333,stroke-width:2px;
    classDef loaderClass fill:#FDE9A9,stroke:#333,stroke-width:2px;
    classDef modelClass fill:#F8C8DC,stroke:#333,stroke-width:2px;
    classDef trainingClass fill:#D1C4E9,stroke:#333,stroke-width:2px;
    classDef vizClass fill:#FFCCBC,stroke:#333,stroke-width:2px;
    classDef utilClass fill:#D7CCC8,stroke:#333,stroke-width:2px;

    %% Click Events for Component Mapping
    click data1 "https://github.com/nivivirat/final-year-project/tree/master/data/"
    click data2 "https://github.com/nivivirat/final-year-project/tree/master/Dataset_Analysis/"
    click data3 "https://github.com/nivivirat/final-year-project/tree/master/indian_data/"
    
    click pre1 "https://github.com/nivivirat/final-year-project/blob/master/ef-sat2rad/Preprocess/save_oneByOne.py"
    click pre2 "https://github.com/nivivirat/final-year-project/blob/master/ef-sat2rad/Preprocess/ef_sat2rad.txt"
    
    click loader "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/earthformer/datasets/"
    click sevir "https://github.com/nivivirat/final-year-project/blob/master/ef-sat2rad/earthformer/datasets/sevir/sevir_dataloader.py"
    
    click cuboid "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/earthformer/cuboid_transformer/"
    click unet "https://github.com/nivivirat/final-year-project/blob/master/ef-sat2rad/model_uNet/UNet3D_baseline.py"
    
    click train "https://github.com/nivivirat/final-year-project/blob/master/ef-sat2rad/train_cuboid_sevir_invLinear.py"
    click experiments "https://github.com/nivivirat/final-year-project/tree/master/experiments/"
    click checkpoint "https://github.com/nivivirat/final-year-project/tree/master/trained_ckpt/"
    
    click metrics "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/earthformer/metrics/"
    click vizModules "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/earthformer/visualization/"
    click viz1 "https://github.com/nivivirat/final-year-project/blob/master/visualize.py"
    click viz2 "https://github.com/nivivirat/final-year-project/blob/master/visualize_output.py"
    
    click utils "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/utils/"
    click earthUtils "https://github.com/nivivirat/final-year-project/tree/master/ef-sat2rad/earthformer/utils/"
```
