# robot-learning-freiburg/pastel architecture

> Architecture diagram of the GitHub repository robot-learning-freiburg/pastel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robot-learning-freiburg/pastel
- Repository: https://github.com/robot-learning-freiburg/pastel

## Overview

##### Project Type & Purpose
- This is a *research/ML pipeline* (not a monolithic web app). It ingests raw images, leverages a pre‐trained Vision Transformer (DINO/ViT), trains two lightweight heads (semantic and boundary), fuses their outputs with normalized‐cut clustering, and optionally runs self‐training on unlabeled images.

a. Configuration Manager
- Loads dataset paths, hyperparameters, pretrained‐weights locations from YAMLs in `src/configs/`
b. Data Layer
- Raw datasets (Cityscapes, PASCAL VOC, PhenoBench)
- `src/datasets` modules: `Dataset` wrapper, transforms
c. Feature Extractor (Foundation Model)
- `src/models/dino_v2.py` or `src/models/vit/...` Vision Transformer backbone
d. Semantic Segmentation Head
- `semantic_fine_tuning.py` invokes Lightning Trainer to fine‐tune a head on extracted features
e. Boundary Estimation Head
- `boundary_fine_tuning.py` similarly fine‐tunes a boundary detection head
f. Fusion & Instance Clustering Module
- `instance_clustering.py`
- Implements normalized‐cut via `src/utils/affinity_matrix.py` & `src/utils/spectral_embedding.py`
- Merges semantic + boundary maps into panoptic pseudo‐labels
g. Self‐Training / Similarity Selection
- `find_similar_images.py` picks unlabeled samples by feature similarity (`src/utils/transforms.py`)
- Loop: generate pseudo‐labels → augment training set → re‐train heads
h. Utilities

##### Relationships & Data Flow
- Outputs feed into Fusion & Clustering
- Clustering → Pseudo‐labels → back into training loop for self‐training
- The Config Manager feeds parameters into all training & clustering steps
- CLI Scripts (`semantic_fine_tuning.py`, `boundary_fine_tuning.py`, `instance_clustering.py`, `find_similar_images.py`) orchestrate each stage

##### Architectural Patterns & Principles
- *Modular pipeline*: each stage is a separate script/module
- *Separation of concerns*: data ingestion, feature extraction, head‐training, clustering, and utilities are in distinct packages
- *Config‐driven*: YAML files abstract dataset‐ and experiment‐specific settings
- *Reusability*: foundation model weights and utility functions shared across stages

##### Essential Elements
- Clear component labels (`Dataset Loader`, `ViT Backbone`, `Semantic Head`, etc.)
- Config block feeding all stages
- Blue for “Data & Config”
- Green for “Model Components”
- Orange for “Clustering & Fusion”
- Gray for “Utilities/Helpers”

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration
    subgraph "Configuration"
        ConfigMgr["Configuration Manager"]:::data
    end

    %% Data Layer
    subgraph "Data & Datasets"
        RawData[(Raw Datasets)]:::data
        subgraph "Data Layer"
            DatasetLoader["Dataset Loader"]:::data
            Cityscapes["Cityscapes Loader"]:::data
            PascalVOC["PASCAL VOC Loader"]:::data
            PhenoBench["PhenoBench Loader"]:::data
        end
    end

    %% Feature Extraction
    subgraph "Feature Extraction"
        ViTBackbone["ViT Backbone"]:::model
        DINO["DINO v2 Model"]:::model
        VITLayers["ViT Layers"]:::model
    end

    %% Fine-Tuning Stage
    subgraph "Fine-Tuning Stage"
        SemanticHead["Semantic Head"]:::model
        BoundaryHead["Boundary Head"]:::model
    end

    %% Fusion & Clustering
    subgraph "Clustering & Fusion"
        InstanceClustering["Fusion & Instance Clustering"]:::fusion
        AffinityMatrix["Affinity Matrix"]:::util
        SpectralEmbedding["Spectral Embedding"]:::util
    end

    %% Self-Training Loop
    subgraph "Self-Training"
        FindSimilar["Similarity Selection"]:::fusion
        SelfTraining["Self-Training Loop"]:::fusion
    end

    %% Utilities
    subgraph "Utilities / Helpers"
        Transforms["Transforms"]:::util
        BoundaryRefine["Boundary Refinement"]:::util
        InstanceColors["Instance Colors"]:::util
    end

    %% External Dependencies
    PretrainedWeights[(Pre-trained Checkpoints)]:::external
    Lightning[(PyTorch Lightning)]:::external

    %% Data & Control Flow
    RawData --> DatasetLoader
    DatasetLoader --> Cityscapes
    DatasetLoader --> PascalVOC
    DatasetLoader --> PhenoBench
    DatasetLoader --> ViTBackbone
    DatasetLoader --> DINO
    DatasetLoader --> VITLayers

    ViTBackbone --> SemanticHead
    DINO --> BoundaryHead
    VITLayers --> SemanticHead
    VITLayers --> BoundaryHead

    SemanticHead --> InstanceClustering
    BoundaryHead --> InstanceClustering

    InstanceClustering --> AffinityMatrix
    InstanceClustering --> SpectralEmbedding
    AffinityMatrix --> InstanceClustering
    SpectralEmbedding --> InstanceClustering

    InstanceClustering --> SelfTraining
    SelfTraining --> FindSimilar
    FindSimilar --> SemanticHead
    FindSimilar --> BoundaryHead

    ConfigMgr --> DatasetLoader
    ConfigMgr --> SemanticHead
    ConfigMgr --> BoundaryHead
    ConfigMgr --> InstanceClustering
    ConfigMgr --> FindSimilar

    PretrainedWeights --> ViTBackbone
    Lightning --> SemanticHead
    Lightning --> BoundaryHead

    Transforms --> DatasetLoader
    Transforms --> FindSimilar
    BoundaryRefine --> InstanceClustering
    InstanceColors --> InstanceClustering

    %% Click Events
    click ConfigMgr "https://github.com/robot-learning-freiburg/pastel/tree/main/src/configs/"
    click Cityscapes "https://github.com/robot-learning-freiburg/pastel/blob/main/src/datasets/cityscapes.py"
    click PascalVOC "https://github.com/robot-learning-freiburg/pastel/blob/main/src/datasets/pascal_voc.py"
    click PhenoBench "https://github.com/robot-learning-freiburg/pastel/blob/main/src/datasets/phenobench.py"
    click DatasetLoader "https://github.com/robot-learning-freiburg/pastel/blob/main/src/datasets/dataset.py"
    click DINO "https://github.com/robot-learning-freiburg/pastel/blob/main/src/models/dino_v2.py"
    click ViTBackbone "https://github.com/robot-learning-freiburg/pastel/blob/main/src/models/vit/vision_transformer.py"
    click VITLayers "https://github.com/robot-learning-freiburg/pastel/tree/main/src/models/vit/layers/"
    click SemanticHead "https://github.com/robot-learning-freiburg/pastel/blob/main/src/semantic_fine_tuning.py"
    click BoundaryHead "https://github.com/robot-learning-freiburg/pastel/blob/main/src/boundary_fine_tuning.py"
    click InstanceClustering "https://github.com/robot-learning-freiburg/pastel/blob/main/src/instance_clustering.py"
    click AffinityMatrix "https://github.com/robot-learning-freiburg/pastel/blob/main/src/utils/affinity_matrix.py"
    click SpectralEmbedding "https://github.com/robot-learning-freiburg/pastel/blob/main/src/utils/spectral_embedding.py"
    click FindSimilar "https://github.com/robot-learning-freiburg/pastel/blob/main/src/find_similar_images.py"
    click Transforms "https://github.com/robot-learning-freiburg/pastel/blob/main/src/utils/transforms.py"
    click BoundaryRefine "https://github.com/robot-learning-freiburg/pastel/blob/main/src/utils/boundary_refinement.py"
    click InstanceColors "https://github.com/robot-learning-freiburg/pastel/blob/main/src/utils/instance_colors.py"

    %% Styles
    classDef data fill:#cde5ff,stroke:#339,stroke-width:1px
    classDef model fill:#d0f0c0,stroke:#393,stroke-width:1px
    classDef fusion fill:#ffe5b4,stroke:#f60,stroke-width:1px
    classDef util fill:#e0e0e0,stroke:#666,stroke-width:1px
    classDef external fill:#f5f5f5,stroke:#999,stroke-width:1px
```
