# jingyechen/mt-transunet architecture

> Architecture diagram of the GitHub repository jingyechen/mt-transunet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jingyechen/mt-transunet
- Repository: https://github.com/jingyechen/mt-transunet

## Overview

##### Project type and purpose
- This is a research‐oriented deep learning pipeline (MT-TransUNet) for joint skin lesion segmentation and classification.
- The code implements data ingestion, model definition (multi‐task Transformer-UNet variants), training/validation loops, and utility modules (losses, metrics, logging, etc.).

##### File structure
- Top-level scripts:
- train.py, engine.py, dataset.py – entry points for data loading, model training and evaluation.
- model/ – core model code, subdivided into:
◦ datasets/ – classes and logic to load image and mask data from disk.
◦ lists/ – text files (“train.txt”, “test_vol.txt”) that enumerate dataset splits.
◦ networks/ – various Transformer-UNet model definitions (vit_seg_*.py, swin, transkip).
◦ train.py, test.py, trainer.py – model training and testing orchestration.
◦ utils.py – model-specific helpers.
- utils/ – shared utilities: learning_rate.py, optimizer.py, loss.py, metrics.py, logger.py, mixup.py, sampler.py, validate.py, runtime.py, prepare.py.

##### Key insights from README
- “MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification.”
- Multi-task design: the model outputs both a segmentation mask and a classification label.

##### Main components and interactions
a. Data Ingestion & Preparation
- Component: Dataset Loader (model/datasets)
- Reads images + masks based on lists in model/lists.
- Applies preprocessing/augmentation via utils/prepare.py & mixup.py.
- Component: DataLoader & Sampler (utils/sampler)
- Wraps datasets into batch iterators.
b. Model
- Component: MT-TransUNet Networks (model/networks)
- Core Transformers + UNet encoder/decoder modules.
- Multi-task heads for segmentation and classification.
c. Training Engine
- Component: Engine (engine.py) & Trainer (model/trainer.py)
- Orchestrates forward pass, loss computation (utils/loss.py), backprop, optimizer step (utils/optimizer.py), learning-rate scheduling (utils/learning_rate.py).
- Records metrics (utils/metrics.py) and logs events (utils/logger.py).
- Component: Validation Loop (utils/validate.py)
- Evaluates on validation split and computes metrics.
d. Testing & Inference
- Component: test.py
- Loads saved model, runs inference on test set, outputs segmentation masks and class predictions.
e. Utilities & Runtime
- Component: runtime, prepare, logger, etc., provide configuration parsing, experiment directory setup and checkpointing.

##### Relationships and data flow
1. Lists → Dataset Loader → Preprocessing/Augmentation → DataLoader/Sampler
2. DataLoader → Training Engine → Model → Loss & Metrics → Optimizer → Model
3. Training Engine → Logger/Checkpoint → Back to DataLoader loop
4. After training: Model → test.py → Inference → Output results

##### Architectural patterns and design principles
- Modular separation of concerns: data, model, training, utilities.
- Pluggable model definitions: each vit_seg_*.py implements a variant.
- Configurable pipelines via script arguments in train.py and engine.py.
- Use of multi-task learning design: shared encoder, separate heads.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Ingestion & Preparation"
        direction TB
        ListsTrain["train.txt"]:::data
        ListsTest["test_vol.txt"]:::data
        ListsAll["all.lst"]:::data
        DataloaderComp["Dataset Loader"]:::data
        Preproc["Preprocessing/Augmentation"]:::data
    end

    subgraph "Data Loader & Sampler"
        direction TB
        DatasetEntry["Custom Dataset Entry Point"]:::data
        Sampler["DataLoader & Sampler"]:::data
    end

    subgraph "Model (MT-TransUNet Networks)"
        direction TB
        Configs["Transformer Configs"]:::model
        BaseModel["Base ViT-UNet"]:::model
        CombineMain["Combine Decoder Variants"]:::model
        CombineAdd["Combine Decoder Add-Image"]:::model
        ResNetSkip["ResNet Skip"]:::model
        ResNetSkipChange["ResNet Skip Change Stem"]:::model
        SwinEnc["Swin Encoder Variant"]:::model
        TransKiPMain["TransKiP Variant"]:::model
        TransKiPAll["TransKiP All"]:::model
    end

    subgraph "Training Orchestration"
        direction TB
        Engine["Engine"]:::orchestration
        TrainScriptTop["Train Script (top-level)"]:::orchestration
        TrainScriptModel["Train Script (model)"]:::orchestration
        Trainer["Trainer Class"]:::orchestration
        Helpers["Model Helpers"]:::orchestration
    end

    subgraph "Loss & Metrics"
        direction TB
        Loss["Loss Functions"]:::utils
        Metrics["Evaluation Metrics"]:::utils
    end

    subgraph "Optimizer & Scheduler"
        direction TB
        Optimizer["Optimizer Setup"]:::utils
        LRScheduler["Learning-Rate Scheduler"]:::utils
    end

    subgraph "Logging & Checkpointing"
        direction TB
        Logger["Logger Utility"]:::utils
        Runtime["Runtime/Experiment Setup"]:::utils
    end

    Validation["Validation Loop"]:::utils

    subgraph "Testing & Inference"
        direction TB
        TestModel["Test Script (model)"]:::inference
        TestGlobal["Test Script (global)"]:::inference
    end

    %% Connections
    ListsTrain --> DataloaderComp
    ListsTest --> DataloaderComp
    ListsAll --> DataloaderComp
    DataloaderComp --> Preproc
    Preproc --> DatasetEntry
    DatasetEntry --> Sampler
    Sampler --> Engine
    TrainScriptTop --> Engine
    TrainScriptModel --> Engine
    Engine --> BaseModel
    BaseModel --> CombineMain
    BaseModel --> CombineAdd
    BaseModel --> ResNetSkip
    BaseModel --> ResNetSkipChange
    BaseModel --> SwinEnc
    BaseModel --> TransKiPMain
    BaseModel --> TransKiPAll
    CombineMain --> Loss
    CombineAdd --> Loss
    ResNetSkip --> Loss
    ResNetSkipChange --> Loss
    SwinEnc --> Loss
    TransKiPMain --> Loss
    TransKiPAll --> Loss
    Loss --> Metrics
    Metrics --> Engine
    Engine --> Optimizer
    Optimizer --> LRScheduler
    LRScheduler --> Engine
    Engine --> Logger
    Logger --> Runtime
    Engine --> Validation
    Engine --> TestModel
    TestModel --> TestGlobal

    %% Click Events
    click DataloaderComp "https://github.com/jingyechen/mt-transunet/blob/main/model/datasets/dataset_synapse.py"
    click ListsTrain "https://github.com/jingyechen/mt-transunet/blob/main/model/lists/lists_Synapse/train.txt"
    click ListsTest "https://github.com/jingyechen/mt-transunet/blob/main/model/lists/lists_Synapse/test_vol.txt"
    click ListsAll "https://github.com/jingyechen/mt-transunet/blob/main/model/lists/lists_Synapse/all.lst"
    click Preproc "https://github.com/jingyechen/mt-transunet/blob/main/utils/prepare.py"
    click Preproc "https://github.com/jingyechen/mt-transunet/blob/main/utils/mixup.py"
    click DatasetEntry "https://github.com/jingyechen/mt-transunet/blob/main/dataset.py"
    click Sampler "https://github.com/jingyechen/mt-transunet/blob/main/utils/sampler.py"
    click Configs "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_configs.py"
    click BaseModel "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling.py"
    click CombineMain "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_combine.py"
    click CombineAdd "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_combine_decoder_add_image.py"
    click ResNetSkip "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_resnet_skip.py"
    click ResNetSkipChange "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_resnet_skip_change_stem.py"
    click SwinEnc "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_swin_encoder.py"
    click TransKiPMain "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_transkip.py"
    click TransKiPAll "https://github.com/jingyechen/mt-transunet/blob/main/model/networks/vit_seg_modeling_transkip_all.py"
    click Engine "https://github.com/jingyechen/mt-transunet/blob/main/engine.py"
    click TrainScriptTop "https://github.com/jingyechen/mt-transunet/blob/main/train.py"
    click TrainScriptModel "https://github.com/jingyechen/mt-transunet/blob/main/model/train.py"
    click Trainer "https://github.com/jingyechen/mt-transunet/blob/main/model/trainer.py"
    click Helpers "https://github.com/jingyechen/mt-transunet/blob/main/model/utils.py"
    click Loss "https://github.com/jingyechen/mt-transunet/blob/main/utils/loss.py"
    click Metrics "https://github.com/jingyechen/mt-transunet/blob/main/utils/metrics.py"
    click Optimizer "https://github.com/jingyechen/mt-transunet/blob/main/utils/optimizer.py"
    click LRScheduler "https://github.com/jingyechen/mt-transunet/blob/main/utils/learning_rate.py"
    click Logger "https://github.com/jingyechen/mt-transunet/blob/main/utils/logger.py"
    click Runtime "https://github.com/jingyechen/mt-transunet/blob/main/utils/runtime.py"
    click Validation "https://github.com/jingyechen/mt-transunet/blob/main/utils/validate.py"
    click TestModel "https://github.com/jingyechen/mt-transunet/blob/main/model/test.py"
    click TestGlobal "https://github.com/jingyechen/mt-transunet/blob/main/test.py"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085;
    classDef model fill:#d4edda,stroke:#155724,color:#155724;
    classDef orchestration fill:#fff3cd,stroke:#856404,color:#856404;
    classDef utils fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef inference fill:#f8d7da,stroke:#721c24,color:#721c24;
```
