# innovator-zero/sak architecture

> Architecture diagram of the GitHub repository innovator-zero/sak, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/innovator-zero/sak
- Repository: https://github.com/innovator-zero/sak

## Overview

##### Project Type and Purpose
- This project is a multi-task learning framework that distills knowledge from multiple vision foundation models. It is a complex deep learning system with separate training stages (Stage 1 for distillation with ImageNet and Stage 2 for multi-task learning with datasets like PASCAL-Context and NYUD-v2).
- Key aspects include teacher-specific adapters, dynamic representation routing (Mixture-of-Representations Routers), and a modular design to preserve the unique representation biases of each foundation model.
- The README emphasizes a teacher–student knowledge distillation setup along with multi-task outputs (e.g., semantic segmentation, boundary detection, depth estimation, etc.).

##### File Structure
- Top-level directories are clearly divided into major modules:
- “configs”: Contains configuration files for stage 1 (s1) and stage 2 (s2) experiments.
- “datasets”: Implements data loading, transformation logic, and dataset-specific utilities for different datasets (ImageNet, NYUD, PASCAL Context, etc.).
- “models”: Organizes the backbone definitions, head modules, and model construction routines, including support for various model families and teacher-specific adapter modules.
- “evaluation”: Contains evaluation scripts and tools for different tasks (edge, depth, semantic segmentation, normals, etc.) including integration with a modified version of SEISM for edge evaluation.
- Utility scripts like train_s1.py, train_s2.py, test.py, and other generic utilities (both for training and testing).
- Additional files such as requirements.txt, LICENSE, and README provide context on dependencies and the project’s academic background.

##### Main Components
a. Training Pipelines:
- Stage 1 Training (train_s1.py): Distillation on ImageNet.
- Stage 2 Training (train_s2.py): Multi-task learning using PASCAL-Context and NYUD-v2.
- Training utilities (train_utils.py) and configuration inputs from the “configs” directory.

b. Data Handling:
- Data loaders in the “datasets” module for ImageNet, NYUD, and PASCAL Context including custom transformations and collate functions.
- The “datasets/utils” folder (e.g., defining paths, configurations) specifies where data is stored and preprocessed.

c. Model Architecture:
- The “models” directory is central. Highlight key elements:
- Backbones (under “models/backbones”) showing various encoder implementations (e.g., HF, TIMM, RADIO, ViT).
- The modular integration: Teacher-Specific Adapter Paths vs. Teacher-Agnostic Stem.
- Model builders, heads, and any distillation-specific modules (e.g., mt_distiller.py).
- The design should capture how multiple backbones are integrated into a unified multi-task network.

d. Evaluation and Testing:
- Scripts under “evaluation” clearly indicate separate evaluation components for different tasks (edge, depth, etc.).
- Connections to third-party tools like the modified SEISM repository for boundary evaluation.

e. Utilities and Miscellaneous:
- General utilities (utils.py) and scripts (test.py) that support training and evaluation.
- Configuration management through YAML files in “configs”.

##### Relationships and Interactions
- How the training pipeline feeds into the model construction process (integrating baseline backbones and adapters) and subsequently outputs checkpoints.
- Evaluation components receiving predictions from the test script and then processing them via evaluation modules (including the MATLAB-based SEISM tool for edge evaluation).
- Two-stage training: Stage 1 distillation followed by Stage 2 multi-task refinement.
- Make it clear how data flows through each module – from reading raw datasets, preprocessing (in the datasets module), using the model for inference, to finally evaluating the performance metrics.

##### Architectural Patterns & Technologies
- Modularity that allows switching out backbones or incorporating new VFMs.
- Point out that it adheres to architectural principles such as decoupling configuration (separate YAML files) from code and enabling scalability (support for distributed training with torchrun).
- Mention specific frameworks used (PyTorch, potential use of WandB for logging) and any specific backends (MATLAB SEISM for edge detection).

- Backbone Modules (“models/backbones”)
- Model Constructors (build_models.py)
- Task-specific Heads and Distillation Modules (mt_distiller.py, heads.py)
- Link Configuration Files from the “configs” directory to both training pipelines and model construction components.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD  
    %% Configuration & Utilities Layer
    subgraph "Configuration & Utilities"
        config1["Configs"]:::config
        config2["train_s1.py"]:::training
        config3["train_s2.py"]:::training
        util1["train_utils.py"]:::training
        util2["utils.py"]:::utility
        projMeta["Project Metadata"]:::utility
    end

    %% Data Handling Layer
    subgraph "Data Handling"
        d1["ImageNet Loader"]:::data
        d2["NYUD Loader"]:::data
        d3["Pascal-Context Loader"]:::data
        d4["Custom Dataset"]:::data
        d5["Custom Transforms"]:::data
        d6["Custom Collate"]:::data
        d7["Dataset Utils"]:::data
    end

    %% Training Pipeline Layer
    subgraph "Training Pipeline"
        t1["Stage 1 Training"]:::training
        t2["Stage 2 Training"]:::training
    end

    %% Model Architecture Layer
    subgraph "Model Architecture"
        m1["HF Encoder"]:::model
        m2["TIMM Encoder"]:::model
        m3["Radio Encoder"]:::model
        m4["ViT Encoder"]:::model
        m5["SAK Encoder"]:::model
        m6["Model Builder"]:::model
        m7["Task-specific Heads"]:::model
        m8["Distillation Module"]:::model
    end

    %% Evaluation & Testing Layer
    subgraph "Evaluation & Testing"
        e1["Test Script"]:::eval
        e2["Evaluation - Depth"]:::eval
        e3["Evaluation - Edge"]:::eval
        e4["Evaluation - HumanParts"]:::eval
        e5["Evaluation - Normals"]:::eval
        e6["Evaluation - Saliency"]:::eval
        e7["Evaluation - SemSeg"]:::eval
        e8["Edge Evaluation (SEISM)"]:::eval
        e9["Evaluation Utilities"]:::eval
        e10["Save Image Utility"]:::eval
    end

    %% Connections: Configuration & Utilities to Training Pipeline
    config1 -->|"provides config"| t1
    config1 -->|"provides config"| t2
    util1 -->|"supports training"| t1
    util1 -->|"supports training"| t2

    %% Connections: Data Handling to Training Pipeline
    d1 -->|"input data"| t1
    d2 -->|"input data"| t2
    d3 -->|"input data"| t2
    d4 -->|"input data"| t1
    d5 -->|"preprocesses"| d1
    d5 -->|"preprocesses"| d2
    d5 -->|"preprocesses"| d3
    d6 -->|"collates data"| d1
    d6 -->|"collates data"| d2
    d6 -->|"collates data"| d3
    d7 -->|"assists data"| d1
    d7 -->|"assists data"| d2
    d7 -->|"assists data"| d3

    %% Connections: Training Pipeline to Model Architecture
    t1 -->|"triggers build"| m6
    t2 -->|"triggers build"| m6
    m6 -->|"integrates"| m1
    m6 -->|"integrates"| m2
    m6 -->|"integrates"| m3
    m6 -->|"integrates"| m4
    m6 -->|"integrates"| m5
    m6 -->|"assembles"| m7
    m6 -->|"assembles"| m8

    %% Connections: Model Architecture to Evaluation & Testing via Test Script
    t1 -->|"produces checkpoint"| e1
    t2 -->|"produces checkpoint"| e1
    m6 -->|"model inference"| e1

    %% Connections: Evaluation & Testing internal flow
    e1 -->|"outputs for depth"| e2
    e1 -->|"outputs for edge"| e3
    e1 -->|"outputs for humanParts"| e4
    e1 -->|"outputs for normals"| e5
    e1 -->|"outputs for saliency"| e6
    e1 -->|"outputs for semseg"| e7
    e3 -->|"enhanced by SEISM"| e8
    e1 -->|"utilizes"| e9
    e1 -->|"utilizes"| e10

    %% Connections: Project Metadata and Utilities
    projMeta --- config1
    projMeta --- util2

    %% Click Events for Configuration & Utilities
    click config1 "https://github.com/innovator-zero/sak/tree/main/configs/"
    click config2 "https://github.com/innovator-zero/sak/blob/main/train_s1.py"
    click config3 "https://github.com/innovator-zero/sak/blob/main/train_s2.py"
    click util1 "https://github.com/innovator-zero/sak/blob/main/train_utils.py"
    click util2 "https://github.com/innovator-zero/sak/blob/main/utils.py"
    click projMeta "https://github.com/innovator-zero/sak/blob/main/README.md"

    %% Click Events for Data Handling
    click d1 "https://github.com/innovator-zero/sak/blob/main/datasets/imagenet.py"
    click d2 "https://github.com/innovator-zero/sak/blob/main/datasets/nyud.py"
    click d3 "https://github.com/innovator-zero/sak/blob/main/datasets/pascal_context.py"
    click d4 "https://github.com/innovator-zero/sak/blob/main/datasets/custom_dataset.py"
    click d5 "https://github.com/innovator-zero/sak/blob/main/datasets/custom_transforms.py"
    click d6 "https://github.com/innovator-zero/sak/blob/main/datasets/utils/custom_collate.py"
    click d7 "https://github.com/innovator-zero/sak/blob/main/datasets/utils/mypath.py"

    %% Click Events for Model Architecture
    click m1 "https://github.com/innovator-zero/sak/blob/main/models/backbones/hf_encoders.py"
    click m2 "https://github.com/innovator-zero/sak/blob/main/models/backbones/timm_encoders.py"
    click m3 "https://github.com/innovator-zero/sak/blob/main/models/backbones/radio.py"
    click m4 "https://github.com/innovator-zero/sak/blob/main/models/backbones/vit.py"
    click m5 "https://github.com/innovator-zero/sak/blob/main/models/backbones/sak.py"
    click m6 "https://github.com/innovator-zero/sak/blob/main/models/build_models.py"
    click m7 "https://github.com/innovator-zero/sak/blob/main/models/heads.py"
    click m8 "https://github.com/innovator-zero/sak/blob/main/models/mt_distiller.py"

    %% Click Events for Evaluation & Testing
    click e1 "https://github.com/innovator-zero/sak/blob/main/test.py"
    click e2 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_depth.py"
    click e3 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_edge.py"
    click e4 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_human_parts.py"
    click e5 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_normals.py"
    click e6 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_sal.py"
    click e7 "https://github.com/innovator-zero/sak/blob/main/evaluation/eval_semseg.py"
    click e8 "https://github.com/innovator-zero/sak/blob/main/evaluation/edge_evaluation.py"
    click e9 "https://github.com/innovator-zero/sak/blob/main/evaluation/evaluate_utils.py"
    click e10 "https://github.com/innovator-zero/sak/blob/main/evaluation/save_img.py"
    click projMeta "https://github.com/innovator-zero/sak/tree/main/LICENSE"
    click projMeta "https://github.com/innovator-zero/sak/blob/main/requirements.txt"

    %% Styles
    classDef config fill:#B3E5FC,stroke:#0288D1,stroke-width:2px;
    classDef data fill:#C8E6C9,stroke:#388E3C,stroke-width:2px;
    classDef training fill:#FFECB3,stroke:#FBC02D,stroke-width:2px;
    classDef model fill:#D1C4E9,stroke:#7B1FA2,stroke-width:2px;
    classDef eval fill:#FFCDD2,stroke:#C62828,stroke-width:2px;
    classDef utility fill:#E0E0E0,stroke:#9E9E9E,stroke-width:2px;
```
