How jingyechen/mt-transunet works
Project type and purpose
• This is a research‐oriented deep learning pipeline (MT-TransUNet) for joint skin lesion segmentation and classification.
• The code implements data ingestion, model definition (multi‐task Transformer-UNet variants), training/validation loops, and utility modules (losses, metrics, logging, etc.).
File structure
• Top-level scripts:
• train.py, engine.py, dataset.py – entry points for data loading, model training and evaluation.
• model/ – core model code, subdivided into:
◦ datasets/ – classes and logic to load image and mask data from disk.
◦ lists/ – text files (“train.txt”, “test_vol.txt”) that enumerate dataset splits.
◦ networks/ – various Transformer-UNet model definitions (vit_seg_*.py, swin, transkip).
◦ train.py, test.py, trainer.py – model training and testing orchestration.
◦ utils.py – model-specific helpers.
• utils/ – shared utilities: learning_rate.py, optimizer.py, loss.py, metrics.py, logger.py, mixup.py, sampler.py, validate.py, runtime.py, prepare.py.
Key insights from README
• “MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification.”
• Multi-task design: the model outputs both a segmentation mask and a classification label.
Main components and interactions
a. Data Ingestion & Preparation
• Component: Dataset Loader (model/datasets)
• Reads images + masks based on lists in model/lists.
• Applies preprocessing/augmentation via utils/prepare.py & mixup.py.
• Component: DataLoader & Sampler (utils/sampler)
• Wraps datasets into batch iterators.
b. Model
• Component: MT-TransUNet Networks (model/networks)
• Core Transformers + UNet encoder/decoder modules.
• Multi-task heads for segmentation and classification.
c. Training Engine
• Component: Engine (engine.py) & Trainer (model/trainer.py)
• Orchestrates forward pass, loss computation (utils/loss.py), backprop, optimizer step (utils/optimizer.py), learning-rate scheduling (utils/learning_rate.py).
• Records metrics (utils/metrics.py) and logs events (utils/logger.py).
• Component: Validation Loop (utils/validate.py)
• Evaluates on validation split and computes metrics.
d. Testing & Inference
• Component: test.py
• Loads saved model, runs inference on test set, outputs segmentation masks and class predictions.
e. Utilities & Runtime
• Component: runtime, prepare, logger, etc., provide configuration parsing, experiment directory setup and checkpointing.
Relationships and data flow
1. Lists → Dataset Loader → Preprocessing/Augmentation → DataLoader/Sampler
2. DataLoader → Training Engine → Model → Loss & Metrics → Optimizer → Model
3. Training Engine → Logger/Checkpoint → Back to DataLoader loop
4. After training: Model → test.py → Inference → Output results
Architectural patterns and design principles
• Modular separation of concerns: data, model, training, utilities.
• Pluggable model definitions: each vit_seg_*.py implements a variant.
• Configurable pipelines via script arguments in train.py and engine.py.
• Use of multi-task learning design: shared encoder, separate heads.