# kavinraj06/segxal-vlm architecture

> Architecture diagram of the GitHub repository kavinraj06/segxal-vlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kavinraj06/segxal-vlm
- Repository: https://github.com/kavinraj06/segxal-vlm

## Overview

1. This repository is a single research pipeline for remote-sensing semantic segmentation on LoveDA, with an active-learning loop and explainability components built around a vision-language segmentation model. The main executable entry points are `main_train.py`, `main_eval.py`, and `main_visualize.py`.

2. Configuration is centralized under `configs/` with separate YAML files for model setup, training hyperparameters, and active-learning policy. This makes the repository mostly config-driven: the runtime behavior is determined by the selected training regime, prompt strategy, and acquisition settings rather than by hardcoded scripts.

3. The data boundary is the `dataset/` package. `loveda.py` likely handles image/mask loading and preprocessing for the LoveDA benchmark, `splits.py` defines train/validation/test or labeled/unlabeled partitions, and `prompts.json` supplies class text prompts used by the language side of the model. The key data flow is remote-sensing image + segmentation label + prompt text.

4. The core model lives in `models/` and is organized into modular encoders and fusion/decoder pieces. `vision_encoder.py` extracts spatial features from imagery, `text_encoder.py` encodes prompts, `prompt_encoder.py` turns semantic prompts into conditioning signals, `cross_attention.py` aligns visual and text features, `decoder.py` produces dense segmentation logits, and `segal_vlm.py` composes these parts into the full SegAL-VLM network.

5. Architecturally, the central computation is multimodal: image features are not segmented in isolation but are conditioned by class semantics from the text prompt stream. Cross-modal attention is the main integration boundary between the vision backbone and the language guidance path, while the decoder is the final boundary that converts fused features into per-pixel class predictions.

6. Training logic is split from the model itself under `training/`. `train_epoch.py` and `validate.py` handle the epoch-level optimization and evaluation cycles, `losses.py` defines segmentation objectives, and `metrics.py` computes quantitative performance such as IoU/mIoU. This separation suggests a conventional supervised training core with reusable evaluation utilities.

7. Active learning is a distinct subsystem under `active_learning/` and is orchestrated by `training/al_loop.py`. `sampler.py` manages pool selection, `dice_selector.py` likely ranks samples by uncertainty or quality signals, and `oracle.py` simulates or abstracts label acquisition from the ground truth pool. The active-learning flow is: train on labeled subset, score unlabeled samples, select new candidates, then expand the labeled set for the next round.

8. Explainability is explicitly modeled, not just visualized afterward. The `xai/` package contains `entropy.py` for uncertainty estimation, `attention_maps.py` for text-guided or cross-attention-based saliency, and `mem.py` for a multimodal explainability module referenced in the README. These components feed the sample-ranking logic and also support post hoc interpretation of why certain regions or images were chosen.

9. Visualization is separated from training and acquisition. `visualization/plot_metrics.py` is for learning-curve and performance plots, while `visualization/visualize_prediction.py` likely renders predicted masks, overlays, and explanation artifacts. This indicates outputs are intended both for quantitative reporting and qualitative inspection.

10. The repository has a clear experimental boundary: it does not appear to expose a service API or application backend. Instead, it is a research codebase focused on offline dataset loading, model training, active-learning iteration, evaluation on LoveDA, and generation of figures/metrics for publication-style analysis.

11. In an architecture diagram, the most important data path is: LoveDA images and class prompts enter the model, vision/text features are fused through cross-attention, segmentation logits and uncertainty/explanation maps are produced, and those outputs drive both evaluation and active-learning selection for the next annotation round.

Main entry points: [`SegAL_VLM_LoveDA/models/segal_vlm.py`](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/segal_vlm.py) (SegAL-VLM), [`SegAL_VLM_LoveDA/main_train.py`](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_train.py) (Train), [`SegAL_VLM_LoveDA/active_learning/dice_selector.py`](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/dice_selector.py) (Dice selector), [`SegAL_VLM_LoveDA/dataset/loveda.py`](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/loveda.py) (LoveDA loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_main_train["Train<br/>entrypoint<br/>[main_train.py]"]
  node_main_eval["Eval<br/>entrypoint<br/>[main_eval.py]"]
  node_main_visualize["Visualize<br/>entrypoint<br/>[main_visualize.py]"]
end

subgraph group_data["Data layer"]
  node_loveda[("LoveDA loader<br/>dataset<br/>[loveda.py]")]
  node_splits["Splits<br/>dataset split<br/>[splits.py]"]
  node_prompts["Prompts<br/>prompt asset<br/>[prompts.json]"]
end

subgraph group_model["Multimodal model"]
  node_vision_encoder["Vision encoder<br/>[vision_encoder.py]"]
  node_text_encoder["Text encoder<br/>[text_encoder.py]"]
  node_prompt_encoder["Prompt encoder<br/>conditioning<br/>[prompt_encoder.py]"]
  node_cross_attention{{"Cross attention<br/>fusion<br/>[cross_attention.py]"}}
  node_decoder["Decoder<br/>segmentation head<br/>[decoder.py]"]
  node_segal_vlm(("SegAL-VLM<br/>multimodal model<br/>[segal_vlm.py]"))
end

subgraph group_train["Training core"]
  node_train_epoch["Train epoch<br/>training loop<br/>[train_epoch.py]"]
  node_validate["Validate<br/>evaluation<br/>[validate.py]"]
  node_losses["Losses<br/>objective<br/>[losses.py]"]
  node_metrics["Metrics<br/>[metrics.py]"]
end

subgraph group_al["Active learning"]
  node_al_loop(("AL loop<br/>orchestrator<br/>[al_loop.py]"))
  node_sampler["Sampler<br/>acquisition<br/>[sampler.py]"]
  node_dice_selector["Dice selector<br/>ranking<br/>[dice_selector.py]"]
  node_oracle["Oracle<br/>label source<br/>[oracle.py]"]
end

subgraph group_xai["Explainability"]
  node_entropy["Entropy<br/>uncertainty<br/>[entropy.py]"]
  node_attention_maps["Attention maps<br/>explanation<br/>[attention_maps.py]"]
  node_mem["MEM<br/>xai module<br/>[mem.py]"]
end

subgraph group_viz["Visualization"]
  node_plot_metrics["Plot metrics<br/>visualization<br/>[plot_metrics.py]"]
  node_visualize_prediction["Predictions<br/>visualization"]
end

subgraph group_cfg["Config"]
  node_model_yaml["Model config<br/>yaml config<br/>[model.yaml]"]
  node_training_yaml["Training config<br/>yaml config<br/>[training.yaml]"]
  node_al_yaml["AL config<br/>yaml config"]
end

node_main_train -->|"loads"| node_model_yaml
node_main_train -->|"loads"| node_training_yaml
node_main_train -->|"loads"| node_al_yaml
node_main_train -->|"uses"| node_loveda
node_main_train -->|"trains"| node_segal_vlm
node_main_eval -->|"evaluates"| node_segal_vlm
node_main_eval -->|"uses"| node_loveda
node_main_visualize -->|"renders"| node_visualize_prediction
node_main_visualize -->|"plots"| node_plot_metrics
node_loveda -->|"partitioned by"| node_splits
node_loveda -->|"paired with"| node_prompts
node_segal_vlm -->|"includes"| node_vision_encoder
node_segal_vlm -->|"includes"| node_text_encoder
node_segal_vlm -->|"includes"| node_prompt_encoder
node_segal_vlm -->|"fuses through"| node_cross_attention
node_segal_vlm -->|"outputs through"| node_decoder
node_prompt_encoder -->|"consumes"| node_prompts
node_text_encoder -->|"consumes"| node_prompts
node_train_epoch -->|"optimizes"| node_losses
node_validate -->|"measures"| node_metrics
node_al_loop -->|"drives"| node_train_epoch
node_al_loop -->|"scores"| node_validate
node_al_loop -->|"selects"| node_sampler
node_sampler -->|"ranks with"| node_dice_selector
node_sampler -->|"queries"| node_oracle
node_dice_selector -->|"uses"| node_entropy
node_dice_selector -->|"uses"| node_attention_maps
node_dice_selector -->|"uses"| node_mem
node_visualize_prediction -->|"overlays"| node_attention_maps
node_visualize_prediction -->|"shows"| node_entropy

click node_main_train "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_train.py"
click node_main_eval "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_eval.py"
click node_main_visualize "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_visualize.py"
click node_loveda "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/loveda.py"
click node_splits "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/splits.py"
click node_prompts "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/prompts.json"
click node_vision_encoder "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/vision_encoder.py"
click node_text_encoder "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/text_encoder.py"
click node_prompt_encoder "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/prompt_encoder.py"
click node_cross_attention "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/cross_attention.py"
click node_decoder "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/decoder.py"
click node_segal_vlm "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/segal_vlm.py"
click node_train_epoch "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/train_epoch.py"
click node_validate "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/validate.py"
click node_losses "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/losses.py"
click node_metrics "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/metrics.py"
click node_al_loop "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/al_loop.py"
click node_sampler "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/sampler.py"
click node_dice_selector "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/dice_selector.py"
click node_oracle "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/oracle.py"
click node_entropy "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/entropy.py"
click node_attention_maps "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/attention_maps.py"
click node_mem "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/mem.py"
click node_plot_metrics "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/visualization/plot_metrics.py"
click node_visualize_prediction "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/visualization/visualize_prediction.py"
click node_model_yaml "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/model.yaml"
click node_training_yaml "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/training.yaml"
click node_al_yaml "https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/active_learning.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_train,node_main_eval,node_main_visualize,node_plot_metrics,node_visualize_prediction toneBlue
class node_loveda,node_splits,node_prompts,node_model_yaml,node_training_yaml,node_al_yaml toneAmber
class node_vision_encoder,node_text_encoder,node_prompt_encoder,node_cross_attention,node_decoder,node_segal_vlm toneMint
class node_train_epoch,node_validate,node_losses,node_metrics toneRose
class node_al_loop,node_sampler,node_dice_selector,node_oracle toneIndigo
class node_entropy,node_attention_maps,node_mem toneTeal
```

## Components

### Entry points

Command-line scripts for train, eval, and visualization.

- [Train](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_train.py): entrypoint · Main training executable. · `SegAL_VLM_LoveDA/main_train.py`
- [Eval](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_eval.py): entrypoint · Main evaluation executable. · `SegAL_VLM_LoveDA/main_eval.py`
- [Visualize](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/main_visualize.py): entrypoint · Main visualization executable. · `SegAL_VLM_LoveDA/main_visualize.py`

### Data layer

LoveDA dataset loading, splits, and prompt assets.

- [LoveDA loader](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/loveda.py): dataset · Image and mask loading plus preprocessing for LoveDA. · `SegAL_VLM_LoveDA/dataset/loveda.py`
- [Splits](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/splits.py): dataset split · Labeled, unlabeled, train, and validation partitions. · `SegAL_VLM_LoveDA/dataset/splits.py`
- [Prompts](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/dataset/prompts.json): prompt asset · Class text prompts used by the language branch. · `SegAL_VLM_LoveDA/dataset/prompts.json`

### Multimodal model

Vision-language segmentation network and its components.

- [Vision encoder](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/vision_encoder.py): encoder · Extracts spatial features from imagery. · `SegAL_VLM_LoveDA/models/vision_encoder.py`
- [Text encoder](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/text_encoder.py): encoder · Encodes class prompts into language features. · `SegAL_VLM_LoveDA/models/text_encoder.py`
- [Prompt encoder](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/prompt_encoder.py): conditioning · Converts semantic prompts into conditioning signals. · `SegAL_VLM_LoveDA/models/prompt_encoder.py`
- [Cross attention](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/cross_attention.py): fusion · Aligns visual and text features across modalities. · `SegAL_VLM_LoveDA/models/cross_attention.py`
- [Decoder](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/decoder.py): segmentation head · Produces dense segmentation logits. · `SegAL_VLM_LoveDA/models/decoder.py`
- [SegAL-VLM](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/models/segal_vlm.py): multimodal model · Composes encoders, fusion, and decoder into the full network. · `SegAL_VLM_LoveDA/models/segal_vlm.py`

### Training core

Optimization, metrics, and evaluation routines.

- [Train epoch](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/train_epoch.py): training loop · Epoch-level optimization step. · `SegAL_VLM_LoveDA/training/train_epoch.py`
- [Validate](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/validate.py): evaluation · Validation and model assessment routine. · `SegAL_VLM_LoveDA/training/validate.py`
- [Losses](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/losses.py): objective · Segmentation loss definitions. · `SegAL_VLM_LoveDA/training/losses.py`
- [Metrics](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/metrics.py): evaluation metrics · IoU and related performance measures. · `SegAL_VLM_LoveDA/training/metrics.py`

### Active learning

Pool sampling and annotation-acquisition loop.

- [AL loop](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/training/al_loop.py): orchestrator · Coordinates training, scoring, and acquisition rounds. · `SegAL_VLM_LoveDA/training/al_loop.py`
- [Sampler](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/sampler.py): acquisition · Selects candidate samples from the pool. · `SegAL_VLM_LoveDA/active_learning/sampler.py`
- [Dice selector](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/dice_selector.py): ranking · Ranks samples using quality or uncertainty signals. · `SegAL_VLM_LoveDA/active_learning/dice_selector.py`
- [Oracle](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/active_learning/oracle.py): label source · Provides or simulates ground-truth acquisition. · `SegAL_VLM_LoveDA/active_learning/oracle.py`

### Explainability

Uncertainty and attention-based explanation modules.

- [Entropy](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/entropy.py): uncertainty · Estimates predictive uncertainty. · `SegAL_VLM_LoveDA/xai/entropy.py`
- [Attention maps](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/attention_maps.py): explanation · Generates attention-based saliency artifacts. · `SegAL_VLM_LoveDA/xai/attention_maps.py`
- [MEM](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/xai/mem.py): xai module · Multimodal explainability module used by the pipeline. · `SegAL_VLM_LoveDA/xai/mem.py`

### Visualization

Metrics and prediction rendering utilities.

- [Plot metrics](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/visualization/plot_metrics.py): visualization · Plots learning curves and performance trends. · `SegAL_VLM_LoveDA/visualization/plot_metrics.py`
- [Predictions](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/visualization/visualize_prediction.py): visualization · Renders predicted masks and explanation overlays. · `SegAL_VLM_LoveDA/visualization/visualize_prediction.py`

### Config

YAML configuration files driving runtime behavior.

- [Model config](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/model.yaml): yaml config · Model and prompt strategy configuration. · `SegAL_VLM_LoveDA/configs/model.yaml`
- [Training config](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/training.yaml): yaml config · Optimization and runtime hyperparameters. · `SegAL_VLM_LoveDA/configs/training.yaml`
- [AL config](https://github.com/kavinraj06/segxal-vlm/blob/main/SegAL_VLM_LoveDA/configs/active_learning.yaml): yaml config · Acquisition policy and active-learning settings. · `SegAL_VLM_LoveDA/configs/active_learning.yaml`

## Connections

- Train → Model config: loads · Training entrypoint reads model configuration.
- Train → Training config: loads · Training entrypoint reads optimization settings.
- Train → AL config: loads · Training entrypoint reads active-learning settings when needed.
- Train → LoveDA loader: uses · Training consumes LoveDA data.
- Train → SegAL-VLM: trains · Main training script drives the multimodal model.
- Eval → SegAL-VLM: evaluates · Evaluation script runs inference on the model.
- Eval → LoveDA loader: uses · Evaluation consumes dataset splits and labels.
- Visualize → Predictions: renders · Visualization entrypoint produces qualitative outputs.
- Visualize → Plot metrics: plots · Visualization entrypoint produces quantitative plots.
- LoveDA loader → Splits: partitioned by · Dataset loading is guided by split definitions.
- LoveDA loader → Prompts: paired with · Image samples are paired with class prompts.
- SegAL-VLM → Vision encoder: includes · Full model contains a vision backbone.
- SegAL-VLM → Text encoder: includes · Full model contains a text encoder.
- SegAL-VLM → Prompt encoder: includes · Full model contains prompt conditioning.
- SegAL-VLM → Cross attention: fuses through · Cross-attention integrates vision and language.
- SegAL-VLM → Decoder: outputs through · Decoder turns fused features into segmentation logits.
- Prompt encoder → Prompts: consumes · Prompt encoding starts from class text prompts.
- Text encoder → Prompts: consumes · Text encoder consumes prompt tokens or text.
- Train epoch → Losses: optimizes · Epoch training uses segmentation objectives.
- Validate → Metrics: measures · Validation computes performance metrics.
- AL loop → Train epoch: drives · Active-learning loop triggers training rounds.
- AL loop → Validate: scores · Active-learning loop evaluates candidate models.
- AL loop → Sampler: selects · Loop delegates pool selection to the sampler.
- Sampler → Dice selector: ranks with · Sampler uses a ranking/scoring policy.
- Sampler → Oracle: queries · Sampler obtains labels from the oracle.
- Dice selector → Entropy: uses · Ranking can use uncertainty estimates.
- Dice selector → Attention maps: uses · Ranking can use attention-based explanations.
- Dice selector → MEM: uses · Ranking can use multimodal explainability signals.
- Predictions → Attention maps: overlays · Prediction visualization can include attention maps.
- Predictions → Entropy: shows · Prediction visualization can show uncertainty.
