How wat-ai/satml works
Project Type & Purpose
• SatML is a machine-learning pipeline for detecting methane plumes in hyperspectral satellite images.
• It encompasses two primary workflows: model training (offline pipeline) and model inference (online or batch prediction).
File Structure
• Top-level directories:
• config: global settings, constants, model hyperparameters (bbox_model_config.yaml, constants.py)
• data: raw hyperspectral TIFFs, CSV manifest, gitignored large files
• models: model definitions (e.g. UNetModule.py), training entry points (train_model.py, train_segmentation.py), model factory
• src: reusable library code (data loader, utilities, loss functions, pipeline orchestration)
• scripts: one-off tools (visualizer, benchmark)
• notebooks: exploratory analysis and prototyping
• logs: runtime logs, checkpoints
• Notice a clear separation between “core library” (src), “model logic” (models), and “orchestration tools” (scripts/notebooks).
Extract Insights from README
• Two PNGs show “Model Training Flow” and “Model Inference Flow.”
• The stack is Python 3.11, with ML frameworks implicitly referenced in requirements (e.g. PyTorch or TensorFlow, NumPy, rasterio).
Main Components & Their Relationships
a. Data Ingestion & Storage
• Raw satellite images (TIFFs) and labels stored under data/raw_data and referenced via train_easy.csv.
• Config files feed parameters into data-loader components.
b. Preprocessing & Utilities
• In src/data_loader.py, src/data_utils.py, src/image_utils.py you standardize, normalize, stack spectral bands.
• File utilities (src/file_utils.py) handle I/O.
c. Model Definition & Training
• Model classes in models/UNetModule.py enable segmentation
• models/cnn_model_factory.py chooses model type
• Training scripts (train_model.py, train_segmentation.py) orchestrate epoch loops, loss computation (from src/losses.py), metrics, checkpointing to logs/.
d. Inference Pipeline
• src/segment_pipeline.py loads trained weights, applies segmentation to new images, writes outputs.
• Visualization or quality-check via scripts/bbox_visualizer.py.
e. External Artifacts & Documentation
• Google Slides plan and research links
• Notebooks for data exploration (notebooks/)
Architectural Patterns & Design Principles
• Modular layering: config → src (core library) → models → scripts
• Factory pattern for model instantiation
• Separation of concerns: training vs. inference vs. experimentation
• Pipeline orchestration (data → preprocess → train/test → evaluate → save artifact)
Final Touches
• Version info (Python 3.11) and major dependencies (e.g. PyTorch, NumPy, rasterio) in a corner.