How robot-learning-freiburg/pastel works
Project Type & Purpose
• This is a *research/ML pipeline* (not a monolithic web app). It ingests raw images, leverages a pre‐trained Vision Transformer (DINO/ViT), trains two lightweight heads (semantic and boundary), fuses their outputs with normalized‐cut clustering, and optionally runs self‐training on unlabeled images.
a. Configuration Manager
• Loads dataset paths, hyperparameters, pretrained‐weights locations from YAMLs in src/configs/
b. Data Layer
• Raw datasets (Cityscapes, PASCAL VOC, PhenoBench)
• src/datasets modules: Dataset wrapper, transforms
c. Feature Extractor (Foundation Model)
• src/models/dino_v2.py or src/models/vit/... Vision Transformer backbone
d. Semantic Segmentation Head
• semantic_fine_tuning.py invokes Lightning Trainer to fine‐tune a head on extracted features
e. Boundary Estimation Head
• boundary_fine_tuning.py similarly fine‐tunes a boundary detection head
f. Fusion & Instance Clustering Module
• instance_clustering.py
• Implements normalized‐cut via src/utils/affinity_matrix.py & src/utils/spectral_embedding.py
• Merges semantic + boundary maps into panoptic pseudo‐labels
g. Self‐Training / Similarity Selection
• find_similar_images.py picks unlabeled samples by feature similarity (src/utils/transforms.py)
• Loop: generate pseudo‐labels → augment training set → re‐train heads
h. Utilities
Relationships & Data Flow
• Outputs feed into Fusion & Clustering
• Clustering → Pseudo‐labels → back into training loop for self‐training
• The Config Manager feeds parameters into all training & clustering steps
• CLI Scripts (semantic_fine_tuning.py, boundary_fine_tuning.py, instance_clustering.py, find_similar_images.py) orchestrate each stage
Architectural Patterns & Principles
• *Modular pipeline*: each stage is a separate script/module
• *Separation of concerns*: data ingestion, feature extraction, head‐training, clustering, and utilities are in distinct packages
• *Config‐driven*: YAML files abstract dataset‐ and experiment‐specific settings
• *Reusability*: foundation model weights and utility functions shared across stages
Essential Elements
• Clear component labels (Dataset Loader, ViT Backbone, Semantic Head, etc.)
• Config block feeding all stages
• Blue for “Data & Config”
• Green for “Model Components”
• Orange for “Clustering & Fusion”
• Gray for “Utilities/Helpers”