# lchen1019/err-seg architecture

> Architecture diagram of the GitHub repository lchen1019/err-seg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lchen1019/err-seg
- Repository: https://github.com/lchen1019/err-seg

## Overview

ERR-Seg is a research implementation of open-vocabulary semantic segmentation: it turns an image and a class vocabulary into pixel-level class predictions, with training and evaluation driven by Detectron2. The main model path reduces image-specific class channels, builds hierarchical image-text costs, aggregates them, and decodes segmentation logits. Dataset registration and label preparation support the documented benchmarks. The excerpts directly support the model, mapper, and training-script connections; some registered-dataset and decoder internals are represented at their exact enclosing paths based on the tree and README.

Main entry points: [`err_seg/modeling/modules/decoder.py`](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/decoder.py) (Cost decoder), [`err_seg/modeling/heads/err_seg_head.py`](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/heads/err_seg_head.py) (Segmentation head), [`train_net.py`](https://github.com/lchen1019/err-seg/blob/main/train_net.py) (Training entry), [`err_seg/err_seg_model.py`](https://github.com/lchen1019/err-seg/blob/main/err_seg/err_seg_model.py) (ERR-Seg model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Dataset input"]
  node_dataset_register["Dataset registration"]
  node_dataset_mapper["Dataset mapping"]
  node_label_prep["Label preparation"]
end

subgraph group_model["Segmentation model"]
  node_err_model["ERR-Seg model<br/>[err_seg_model.py]"]
  node_head["Segmentation head<br/>[err_seg_head.py]"]
  node_clip["OpenCLIP model<br/>[factory.py]"]
  node_tokenizer["Text tokenizer<br/>[tokenizer.py]"]
  node_transformer["Text transformer<br/>[transformer.py]"]
  node_decoder["Cost decoder<br/>[decoder.py]"]
  node_fusion["Feature fusion<br/>[fusion.py]"]
  node_class_fusion["Class fusion<br/>[class_fusion.py]"]
  node_spatial_fusion["Spatial fusion<br/>[spatial_fusion.py]"]
  node_aggregator["Cost aggregation<br/>[aggregator.py]"]
  node_up_decoder["Upsampling decoder<br/>[up_decoder.py]"]
end

subgraph group_runtime["Train and evaluate"]
  node_train_net["Training entry<br/>[train_net.py]"]
  node_prediction["Pixel predictions<br/>[err_seg_model.py]"]
  node_train_data["Training data<br/>[train_net.py]"]
  node_evaluation["Evaluation<br/>[train_net.py]"]
end

node_user(("Research user"))
node_images[("Images and labels")]
node_vocab["Class vocabulary"]

node_user -->|"starts run"| node_train_net
node_images -->|"provides data"| node_dataset_register
node_label_prep -->|"writes labels"| node_dataset_register
node_dataset_register -->|"registers datasets"| node_train_net
node_train_net -->|"builds loader"| node_dataset_mapper
node_dataset_mapper -->|"maps samples"| node_train_data
node_train_net -->|"builds model"| node_err_model
node_train_data -->|"supplies batches"| node_err_model
node_err_model -->|"forwards features"| node_head
node_vocab -->|"defines classes"| node_head
node_head -->|"creates model"| node_clip
node_head -->|"tokenizes prompts"| node_tokenizer
node_clip -->|"encodes text"| node_transformer
node_head -->|"passes features"| node_decoder
node_decoder -->|"fuses costs"| node_fusion
node_decoder -->|"reduces classes"| node_class_fusion
node_decoder -->|"reduces spatial tokens"| node_spatial_fusion
node_decoder -->|"aggregates costs"| node_aggregator
node_decoder -->|"decodes features"| node_up_decoder
node_err_model -->|"postprocesses logits"| node_prediction
node_prediction -->|"scores predictions"| node_evaluation
node_train_net -->|"runs evaluation"| node_evaluation

click node_train_net "https://github.com/lchen1019/err-seg/blob/main/train_net.py"
click node_dataset_register "https://github.com/lchen1019/err-seg/tree/main/err_seg/data/datasets"
click node_dataset_mapper "https://github.com/lchen1019/err-seg/blob/main/err_seg/data/dataset_mappers/mask_former_semantic_dataset_mapper.py"
click node_label_prep "https://github.com/lchen1019/err-seg/tree/main/datasets"
click node_err_model "https://github.com/lchen1019/err-seg/blob/main/err_seg/err_seg_model.py"
click node_head "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/heads/err_seg_head.py"
click node_clip "https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/factory.py"
click node_tokenizer "https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/tokenizer.py"
click node_transformer "https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/transformer.py"
click node_decoder "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/decoder.py"
click node_fusion "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/fusion.py"
click node_class_fusion "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/class_fusion.py"
click node_spatial_fusion "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/spatial_fusion.py"
click node_aggregator "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/aggregator.py"
click node_up_decoder "https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/up_decoder.py"
click node_prediction "https://github.com/lchen1019/err-seg/blob/main/err_seg/err_seg_model.py"
click node_train_data "https://github.com/lchen1019/err-seg/blob/main/train_net.py"
click node_evaluation "https://github.com/lchen1019/err-seg/blob/main/train_net.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_register,node_dataset_mapper,node_label_prep,node_user toneBlue
class node_err_model,node_head,node_clip,node_tokenizer,node_transformer,node_decoder,node_fusion,node_class_fusion,node_spatial_fusion,node_aggregator,node_up_decoder,node_images toneAmber
class node_train_net,node_prediction,node_train_data,node_evaluation toneMint
class node_vocab toneIndigo
```

## Components

### Dataset input

- [Dataset registration](https://github.com/lchen1019/err-seg/tree/main/err_seg/data/datasets): `err_seg/data/datasets`
- [Dataset mapping](https://github.com/lchen1019/err-seg/blob/main/err_seg/data/dataset_mappers/mask_former_semantic_dataset_mapper.py): `err_seg/data/dataset_mappers/mask_former_semantic_dataset_mapper.py`
- [Label preparation](https://github.com/lchen1019/err-seg/tree/main/datasets): `datasets`

### Segmentation model

- [ERR-Seg model](https://github.com/lchen1019/err-seg/blob/main/err_seg/err_seg_model.py): `err_seg/err_seg_model.py`
- [Segmentation head](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/heads/err_seg_head.py): `err_seg/modeling/heads/err_seg_head.py`
- [OpenCLIP model](https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/factory.py): `open_clip/src/open_clip/factory.py`
- [Text tokenizer](https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/tokenizer.py): `open_clip/src/open_clip/tokenizer.py`
- [Text transformer](https://github.com/lchen1019/err-seg/blob/main/open_clip/src/open_clip/transformer.py): `open_clip/src/open_clip/transformer.py`
- [Cost decoder](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/decoder.py): `err_seg/modeling/modules/decoder.py`
- [Feature fusion](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/fusion.py): `err_seg/modeling/modules/fusion.py`
- [Class fusion](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/class_fusion.py): `err_seg/modeling/modules/class_fusion.py`
- [Spatial fusion](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/spatial_fusion.py): `err_seg/modeling/modules/spatial_fusion.py`
- [Cost aggregation](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/aggregator.py): `err_seg/modeling/modules/aggregator.py`
- [Upsampling decoder](https://github.com/lchen1019/err-seg/blob/main/err_seg/modeling/modules/up_decoder.py): `err_seg/modeling/modules/up_decoder.py`

### Train and evaluate

- [Training entry](https://github.com/lchen1019/err-seg/blob/main/train_net.py): `train_net.py`
- [Pixel predictions](https://github.com/lchen1019/err-seg/blob/main/err_seg/err_seg_model.py): `err_seg/err_seg_model.py`
- [Training data](https://github.com/lchen1019/err-seg/blob/main/train_net.py): `train_net.py`
- [Evaluation](https://github.com/lchen1019/err-seg/blob/main/train_net.py): `train_net.py`

### Other components

- **Research user**
- **Images and labels**
- **Class vocabulary**

## Connections

- Research user → Training entry: starts run
- Images and labels → Dataset registration: provides data
- Label preparation → Dataset registration: writes labels
- Dataset registration → Training entry: registers datasets
- Training entry → Dataset mapping: builds loader
- Dataset mapping → Training data: maps samples
- Training entry → ERR-Seg model: builds model
- Training data → ERR-Seg model: supplies batches
- ERR-Seg model → Segmentation head: forwards features
- Class vocabulary → Segmentation head: defines classes
- Segmentation head → OpenCLIP model: creates model
- Segmentation head → Text tokenizer: tokenizes prompts
- OpenCLIP model → Text transformer: encodes text
- Segmentation head → Cost decoder: passes features
- Cost decoder → Feature fusion: fuses costs
- Cost decoder → Class fusion: reduces classes
- Cost decoder → Spatial fusion: reduces spatial tokens
- Cost decoder → Cost aggregation: aggregates costs
- Cost decoder → Upsampling decoder: decodes features
- ERR-Seg model → Pixel predictions: postprocesses logits
- Pixel predictions → Evaluation: scores predictions
- Training entry → Evaluation: runs evaluation