How kavinraj06/segxal-vlm works
1. This repository is a single research pipeline for remote-sensing semantic segmentation on LoveDA, with an active-learning loop and explainability components built around a vision-language segmentation model. The main executable entry points are main_train.py, main_eval.py, and main_visualize.py.
2. Configuration is centralized under configs/ with separate YAML files for model setup, training hyperparameters, and active-learning policy. This makes the repository mostly config-driven: the runtime behavior is determined by the selected training regime, prompt strategy, and acquisition settings rather than by hardcoded scripts.
3. The data boundary is the dataset/ package. loveda.py likely handles image/mask loading and preprocessing for the LoveDA benchmark, splits.py defines train/validation/test or labeled/unlabeled partitions, and prompts.json supplies class text prompts used by the language side of the model. The key data flow is remote-sensing image + segmentation label + prompt text.
4. The core model lives in models/ and is organized into modular encoders and fusion/decoder pieces. vision_encoder.py extracts spatial features from imagery, text_encoder.py encodes prompts, prompt_encoder.py turns semantic prompts into conditioning signals, cross_attention.py aligns visual and text features, decoder.py produces dense segmentation logits, and segal_vlm.py composes these parts into the full SegAL-VLM network.
5. Architecturally, the central computation is multimodal: image features are not segmented in isolation but are conditioned by class semantics from the text prompt stream. Cross-modal attention is the main integration boundary between the vision backbone and the language guidance path, while the decoder is the final boundary that converts fused features into per-pixel class predictions.
6. Training logic is split from the model itself under training/. train_epoch.py and validate.py handle the epoch-level optimization and evaluation cycles, losses.py defines segmentation objectives, and metrics.py computes quantitative performance such as IoU/mIoU. This separation suggests a conventional supervised training core with reusable evaluation utilities.
7. Active learning is a distinct subsystem under active_learning/ and is orchestrated by training/al_loop.py. sampler.py manages pool selection, dice_selector.py likely ranks samples by uncertainty or quality signals, and oracle.py simulates or abstracts label acquisition from the ground truth pool. The active-learning flow is: train on labeled subset, score unlabeled samples, select new candidates, then expand the labeled set for the next round.
8. Explainability is explicitly modeled, not just visualized afterward. The xai/ package contains entropy.py for uncertainty estimation, attention_maps.py for text-guided or cross-attention-based saliency, and mem.py for a multimodal explainability module referenced in the README. These components feed the sample-ranking logic and also support post hoc interpretation of why certain regions or images were chosen.
9. Visualization is separated from training and acquisition. visualization/plot_metrics.py is for learning-curve and performance plots, while visualization/visualize_prediction.py likely renders predicted masks, overlays, and explanation artifacts. This indicates outputs are intended both for quantitative reporting and qualitative inspection.
10. The repository has a clear experimental boundary: it does not appear to expose a service API or application backend. Instead, it is a research codebase focused on offline dataset loading, model training, active-learning iteration, evaluation on LoveDA, and generation of figures/metrics for publication-style analysis.
11. In an architecture diagram, the most important data path is: LoveDA images and class prompts enter the model, vision/text features are fused through cross-attention, segmentation logits and uncertainty/explanation maps are produced, and those outputs drive both evaluation and active-learning selection for the next annotation round.
Main entry points: segal_vlm.py SegAL-VLM · main_train.py Train · dice_selector.py Dice selector · loveda.py LoveDA loader
Components 28
Entry points
Command-line scripts for train, eval, and visualization.
Data layer
LoveDA dataset loading, splits, and prompt assets.
- LoveDA loader · Image and mask loading plus preprocessing for LoveDA.
SegAL_VLM_LoveDA/dataset/loveda.py - Splits · Labeled, unlabeled, train, and validation partitions.
SegAL_VLM_LoveDA/dataset/splits.py - Prompts · Class text prompts used by the language branch.
SegAL_VLM_LoveDA/dataset/prompts.json
Multimodal model
Vision-language segmentation network and its components.
- Vision encoder · Extracts spatial features from imagery.
SegAL_VLM_LoveDA/models/vision_encoder.py - Text encoder · Encodes class prompts into language features.
SegAL_VLM_LoveDA/models/text_encoder.py - Prompt encoder · Converts semantic prompts into conditioning signals.
SegAL_VLM_LoveDA/models/prompt_encoder.py - Cross attention · Aligns visual and text features across modalities.
SegAL_VLM_LoveDA/models/cross_attention.py - Decoder · Produces dense segmentation logits.
SegAL_VLM_LoveDA/models/decoder.py - SegAL-VLM · Composes encoders, fusion, and decoder into the full network.
SegAL_VLM_LoveDA/models/segal_vlm.py
Training core
Optimization, metrics, and evaluation routines.
- Train epoch · Epoch-level optimization step.
SegAL_VLM_LoveDA/training/train_epoch.py - Validate · Validation and model assessment routine.
SegAL_VLM_LoveDA/training/validate.py - Losses · Segmentation loss definitions.
SegAL_VLM_LoveDA/training/losses.py - Metrics · IoU and related performance measures.
SegAL_VLM_LoveDA/training/metrics.py
Active learning
Pool sampling and annotation-acquisition loop.
- AL loop · Coordinates training, scoring, and acquisition rounds.
SegAL_VLM_LoveDA/training/al_loop.py - Sampler · Selects candidate samples from the pool.
SegAL_VLM_LoveDA/active_learning/sampler.py - Dice selector · Ranks samples using quality or uncertainty signals.
SegAL_VLM_LoveDA/active_learning/dice_selector.py - Oracle · Provides or simulates ground-truth acquisition.
SegAL_VLM_LoveDA/active_learning/oracle.py
Explainability
Uncertainty and attention-based explanation modules.
- Entropy · Estimates predictive uncertainty.
SegAL_VLM_LoveDA/xai/entropy.py - Attention maps · Generates attention-based saliency artifacts.
SegAL_VLM_LoveDA/xai/attention_maps.py - MEM · Multimodal explainability module used by the pipeline.
SegAL_VLM_LoveDA/xai/mem.py
Visualization
Metrics and prediction rendering utilities.
- Plot metrics · Plots learning curves and performance trends.
SegAL_VLM_LoveDA/visualization/plot_metrics.py - Predictions · Renders predicted masks and explanation overlays.
SegAL_VLM_LoveDA/visualization/visualize_prediction.py
Config
YAML configuration files driving runtime behavior.
- Model config · Model and prompt strategy configuration.
SegAL_VLM_LoveDA/configs/model.yaml - Training config · Optimization and runtime hyperparameters.
SegAL_VLM_LoveDA/configs/training.yaml - AL config · Acquisition policy and active-learning settings.
SegAL_VLM_LoveDA/configs/active_learning.yaml
Connections 30
- Train → Model config: loads · Training entrypoint reads model configuration.
- Train → Training config: loads · Training entrypoint reads optimization settings.
- Train → AL config: loads · Training entrypoint reads active-learning settings when needed.
- Train → LoveDA loader: uses · Training consumes LoveDA data.
- Train → SegAL-VLM: trains · Main training script drives the multimodal model.
- Eval → SegAL-VLM: evaluates · Evaluation script runs inference on the model.
- Eval → LoveDA loader: uses · Evaluation consumes dataset splits and labels.
- Visualize → Predictions: renders · Visualization entrypoint produces qualitative outputs.
- Visualize → Plot metrics: plots · Visualization entrypoint produces quantitative plots.
- LoveDA loader → Splits: partitioned by · Dataset loading is guided by split definitions.
- LoveDA loader → Prompts: paired with · Image samples are paired with class prompts.
- SegAL-VLM → Vision encoder: includes · Full model contains a vision backbone.
- SegAL-VLM → Text encoder: includes · Full model contains a text encoder.
- SegAL-VLM → Prompt encoder: includes · Full model contains prompt conditioning.
- SegAL-VLM → Cross attention: fuses through · Cross-attention integrates vision and language.
- SegAL-VLM → Decoder: outputs through · Decoder turns fused features into segmentation logits.
- Prompt encoder → Prompts: consumes · Prompt encoding starts from class text prompts.
- Text encoder → Prompts: consumes · Text encoder consumes prompt tokens or text.
- Train epoch → Losses: optimizes · Epoch training uses segmentation objectives.
- Validate → Metrics: measures · Validation computes performance metrics.
- AL loop → Train epoch: drives · Active-learning loop triggers training rounds.
- AL loop → Validate: scores · Active-learning loop evaluates candidate models.
- AL loop → Sampler: selects · Loop delegates pool selection to the sampler.
- Sampler → Dice selector: ranks with · Sampler uses a ranking/scoring policy.
- Sampler → Oracle: queries · Sampler obtains labels from the oracle.
- Dice selector → Entropy: uses · Ranking can use uncertainty estimates.
- Dice selector → Attention maps: uses · Ranking can use attention-based explanations.
- Dice selector → MEM: uses · Ranking can use multimodal explainability signals.
- Predictions → Attention maps: overlays · Prediction visualization can include attention maps.
- Predictions → Entropy: shows · Prediction visualization can show uncertainty.