# bowang-lab/medsam2 architecture

> Architecture diagram of the GitHub repository bowang-lab/medsam2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bowang-lab/medsam2
- Repository: https://github.com/bowang-lab/medsam2

## Overview

##### Project Type and Purpose
- MedSAM2 is a Python-based research toolkit and application library for “Segment Anything” in 3D medical images and videos.
- It provides:
- A core model implementation (based on the SAM2 architecture) for 3D medical image and video segmentation.
- Training pipelines (single-node and multi-node).
- Inference entrypoints via CLI scripts, Jupyter notebooks, and a Gradio demo app.

##### File Structure Analysis
- Top-level:
- app.py – Gradio demo server
- medsam2_infer_3D_CT.py, medsam2_infer_video.py – CLI inference scripts
- multi_node_train.sh, training/train.py – distributed training scripts
- notebooks/, examples/ – interactive examples and tutorials
- checkpoints/ – pre-trained model files
- sam2/ – core library implementing model building, encoders, decoders, predictors
- training/ – data loaders, loss functions, training utilities, model definition for training
- sam2/configs/ – YAML config files for model hyperparameters
- sam2/csrc/ – CUDA kernels for connected components
- pyproject.toml, setup.py – packaging and dependency specs

##### README Insights
- Confirms three main use cases: 3D CT segmentation, video segmentation, and Gradio demo.
- Emphasizes downloading checkpoints, installing PyTorch and Gradio, running training and inference commands.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Entrypoints / Interfaces"
        CLI3D["CLI Inference 3D CT"]:::interfaces
        click CLI3D "https://github.com/bowang-lab/medsam2/blob/main/medsam2_infer_3D_CT.py"
        CLIVid["CLI Inference Video"]:::interfaces
        click CLIVid "https://github.com/bowang-lab/medsam2/blob/main/medsam2_infer_video.py"
        TrainSN["Single-node Training"]:::interfaces
        click TrainSN "https://github.com/bowang-lab/medsam2/blob/main/training/train.py"
        TrainMN["Multi-node Training Launcher"]:::interfaces
        click TrainMN "https://github.com/bowang-lab/medsam2/blob/main/multi_node_train.sh"
        Notebooks["Jupyter Notebooks"]:::interfaces
        click Notebooks "https://github.com/bowang-lab/medsam2/tree/main/notebooks/"
        Gradio["Gradio Demo App"]:::interfaces
        click Gradio "https://github.com/bowang-lab/medsam2/blob/main/app.py"
    end

    subgraph "Core Library (sam2)"
        Builder["SAM2 Model Builder"]:::core
        click Builder "https://github.com/bowang-lab/medsam2/blob/main/sam2/build_sam.py"
        Config1["Config: sam2.1_hiera_t512.yaml"]:::core
        click Config1 "https://github.com/bowang-lab/medsam2/blob/main/sam2/configs/sam2.1_hiera_t512.yaml"
        Config2["Config: sam2.1_hiera_tiny_finetune512.yaml"]:::core
        click Config2 "https://github.com/bowang-lab/medsam2/blob/main/sam2/configs/sam2.1_hiera_tiny_finetune512.yaml"
        CUDACore["CUDA Kernels"]:::core
        click CUDACore "https://github.com/bowang-lab/medsam2/blob/main/sam2/csrc/connected_components.cu"
        Modeling["Modeling (Encoders/Decoders/Attention)"]:::core
        click Modeling "https://github.com/bowang-lab/medsam2/tree/main/sam2/modeling/"
        ImgPred["Image Predictor API"]:::core
        click ImgPred "https://github.com/bowang-lab/medsam2/blob/main/sam2/sam2_image_predictor.py"
        VidPred["Video Predictor API"]:::core
        click VidPred "https://github.com/bowang-lab/medsam2/blob/main/sam2/sam2_video_predictor.py"
        VidPred2["Video Predictor NPZ"]:::core
        click VidPred2 "https://github.com/bowang-lab/medsam2/blob/main/sam2/sam2_video_predictor_npz.py"
        VidTrainer["Video Trainer API"]:::core
        click VidTrainer "https://github.com/bowang-lab/medsam2/blob/main/sam2/sam2_video_trainer.py"
        UtilsCore["Utilities"]:::core
        click UtilsCore "https://github.com/bowang-lab/medsam2/tree/main/sam2/utils/"
    end

    subgraph "Training Pipeline (training)"
        DataLoader["Dataset Loaders & Samplers"]:::training
        click DataLoader "https://github.com/bowang-lab/medsam2/tree/main/training/dataset/"
        Loss["Loss Functions"]:::training
        click Loss "https://github.com/bowang-lab/medsam2/blob/main/training/loss_fns.py"
        Opt["Optimizer & Scheduler"]:::training
        click Opt "https://github.com/bowang-lab/medsam2/blob/main/training/optimizer.py"
        Trainer["Trainer Orchestration"]:::training
        click Trainer "https://github.com/bowang-lab/medsam2/blob/main/training/trainer.py"
        Submitit["Submitit Scripts"]:::training
        click Submitit "https://github.com/bowang-lab/medsam2/blob/main/training/scripts/sav_frame_extraction_submitit.py"
        CUt1["checkpoint_utils.py"]:::training
        click CUt1 "https://github.com/bowang-lab/medsam2/blob/main/training/utils/checkpoint_utils.py"
        CUt2["logger.py"]:::training
        click CUt2 "https://github.com/bowang-lab/medsam2/blob/main/training/utils/logger.py"
        CUt3["distributed.py"]:::training
        click CUt3 "https://github.com/bowang-lab/medsam2/blob/main/training/utils/distributed.py"
        CUt4["train_utils.py"]:::training
        click CUt4 "https://github.com/bowang-lab/medsam2/blob/main/training/utils/train_utils.py"
    end

    subgraph "Data Stores"
        RawData["Raw Images/Videos"]:::datastore
        CheckpointsStore["Pre-trained Weights Store"]:::datastore
        click CheckpointsStore "https://github.com/bowang-lab/medsam2/tree/main/checkpoints/"
        Outputs["Segmentation Outputs"]:::datastore
    end

    subgraph "External Dependencies"
        PyTorch["PyTorch"]:::external
        CUDA["CUDA"]:::external
        FFMPEG["ffmpeg/moviepy"]:::external
        GrDep["Gradio"]:::external
        SubmititDep["submitit/SLURM"]:::external
    end

    %% Data Flows - Inference
    RawData -->|"input data"| ImgPred
    ImgPred -->|"segmentation output"| Outputs
    RawData -->|"input data"| VidPred
    VidPred -->|"segmentation output"| Outputs
    FFMPEG --> VidPred
    Gradio --> ImgPred
    Gradio --> VidPred

    CLI3D --> ImgPred
    CLIVid --> VidPred
    Notebooks --> Builder

    %% Data Flows - Training
    RawData --> DataLoader
    DataLoader --> Builder
    Builder --> Loss
    Loss --> Opt
    Opt --> Trainer
    Trainer --> CUt1
    CUt1 --> CheckpointsStore
    Trainer --> CUt2
    Trainer --> CUt3
    Trainer --> CUt4
    Submitit --> SubmititDep

    TrainSN --> Trainer
    TrainMN --> Submitit

    %% Core library dependencies
    Config1 --> Builder
    Config2 --> Builder
    CUDACore --> Modeling
    Modeling --> Builder
    PyTorch --> Modeling
    CUDA --> Modeling
    PyTorch --> DataLoader
    PyTorch --> Opt
    CUDA --> SubmititDep

    %% Styles
    classDef interfaces fill:#D6EAF8,stroke:#3498DB,color:#1B4F72
    classDef core fill:#E8F8F5,stroke:#1ABC9C,color:#0E6251
    classDef training fill:#FEF9E7,stroke:#F1C40F,color:#7D6608
    classDef datastore fill:#F5EEF8,stroke:#8E44AD,color:#512E5F
    classDef external fill:#F4F6F6,stroke:#566573,color:#2C3E50
```
