How lchen1019/err-seg works
ERR-Seg is a research implementation of open-vocabulary semantic segmentation: it turns an image and a class vocabulary into pixel-level class predictions, with training and evaluation driven by Detectron2. The main model path reduces image-specific class channels, builds hierarchical image-text costs, aggregates them, and decodes segmentation logits. Dataset registration and label preparation support the documented benchmarks. The excerpts directly support the model, mapper, and training-script connections; some registered-dataset and decoder internals are represented at their exact enclosing paths based on the tree and README.
Main entry points: decoder.py Cost decoder · err_seg_head.py Segmentation head · train_net.py Training entry · err_seg_model.py ERR-Seg model
Components 21
Dataset input
- Dataset registration
err_seg/data/datasets - Dataset mapping
err_seg/data/dataset_mappers/mask_former_semantic_dataset_mapper.py - Label preparation
datasets
Segmentation model
- ERR-Seg model
err_seg/err_seg_model.py - Segmentation head
err_seg/modeling/heads/err_seg_head.py - OpenCLIP model
open_clip/src/open_clip/factory.py - Text tokenizer
open_clip/src/open_clip/tokenizer.py - Text transformer
open_clip/src/open_clip/transformer.py - Cost decoder
err_seg/modeling/modules/decoder.py - Feature fusion
err_seg/modeling/modules/fusion.py - Class fusion
err_seg/modeling/modules/class_fusion.py - Spatial fusion
err_seg/modeling/modules/spatial_fusion.py - Cost aggregation
err_seg/modeling/modules/aggregator.py - Upsampling decoder
err_seg/modeling/modules/up_decoder.py
Train and evaluate
- Training entry
train_net.py - Pixel predictions
err_seg/err_seg_model.py - Training data
train_net.py - Evaluation
train_net.py
Other components
- Research user
- Images and labels
- Class vocabulary
Connections 22
- Research user → Training entry: starts run
- Images and labels → Dataset registration: provides data
- Label preparation → Dataset registration: writes labels
- Dataset registration → Training entry: registers datasets
- Training entry → Dataset mapping: builds loader
- Dataset mapping → Training data: maps samples
- Training entry → ERR-Seg model: builds model
- Training data → ERR-Seg model: supplies batches
- ERR-Seg model → Segmentation head: forwards features
- Class vocabulary → Segmentation head: defines classes
- Segmentation head → OpenCLIP model: creates model
- Segmentation head → Text tokenizer: tokenizes prompts
- OpenCLIP model → Text transformer: encodes text
- Segmentation head → Cost decoder: passes features
- Cost decoder → Feature fusion: fuses costs
- Cost decoder → Class fusion: reduces classes
- Cost decoder → Spatial fusion: reduces spatial tokens
- Cost decoder → Cost aggregation: aggregates costs
- Cost decoder → Upsampling decoder: decodes features
- ERR-Seg model → Pixel predictions: postprocesses logits
- Pixel predictions → Evaluation: scores predictions
- Training entry → Evaluation: runs evaluation