How nivivirat/final-year-project works
Main Components
a. Data Sources and Storage:
• Input data is stored in multiple HDF5 (.h5) files. There are directories like “data”, “Dataset_Analysis”, and “indian_data” that hold different datasets (e.g., SEVIR, Indian data).
• These datasets serve as the raw inputs for training and testing.
b. Preprocessing:
• Under the “ef-sat2rad/Preprocess” directory, scripts (such as save_oneByOne.py and ef_sat2rad.txt) indicate that data normalization, transformation, or augmentation is performed before training.
c. Core Model Components:
• The “ef-sat2rad/earthformer” directory contains critical modules:
• “cuboid_transformer”: likely handling spatial-temporal transformations.
• “datasets”, “metrics”, and “visualization”: responsible for loading data, computing performance metrics (such as enso, skill scores, etc.), and visualizing outcomes.
• The “ef-sat2rad/model_uNet” directory includes a U-Net variant (UNet3D_baseline.py) that might be used for segmentation or spatial prediction tasks.
d. Training and Experimentation:
• The script “train_cuboid_sevir_invLinear.py” acts as the training entry point.
• The “experiments” directory manages configuration files, logging outputs (with several “lightning_logs” versions), and saved intermediate outputs.
• The “trained_ckpt” folder stores the trained model checkpoints.
e. Auxiliary Components:
• Utility modules in both “ef-sat2rad/utils” and various “earthformer/utils” help with optimization, checkpoint handling, and other supportive functions.
Relationships and Data Flows
• Data → Model Pipeline: From the data loaders, the data is input to the core models – the cuboid transformer (and associated baselines) as well as the U-Net model; indicate data flow to these components.
• Model Predictions → Visualization: Finally, outputs progress to “visualization” modules (across “ef-sat2rad/earthformer/visualization” and separate visualizer scripts) that help interpret the results.
Architectural Patterns and Design Principles
• Modular Design: The separation between preprocessing, model definition (earthformer and model_uNet), training, and evaluation/logging is key.
• Pipeline-Oriented Workflow: Data flows sequentially from raw/preprocessed inputs, through data augmentation, model training, and finally evaluation/visualization.
• Configurability: The presence of YAML configuration files (like “ef-sat2rad/cfgs/cfg.yaml” and “experiments/cfg.yaml”) suggests that parameters and model settings are externalized, emphasizing flexibility.
• Scalability and Reproducibility: Experiment logs and checkpointing indicate a focus on reproducible experiments.
Relevant Technologies/Frameworks
• Deep Learning Framework: Likely PyTorch (or a derivative) is being used, as indicated by “lightning_logs” (possibly via PyTorch Lightning) and modules like “apex_ddp.py” for distributed training.
• Data Formats: HDF5 files are used extensively.
• Configuration & Environment: The use of Conda (environment.yml and conda commands) points to dependency management best practices.