# bowang-lab/medsam architecture

> Architecture diagram of the GitHub repository bowang-lab/medsam, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bowang-lab/medsam
- Repository: https://github.com/bowang-lab/medsam

## Overview

##### Project type and purpose
- MedSAM is a full-stack research toolkit for 2D/3D medical image segmentation.
- Purpose: fine-tune and extend the “Segment Anything” (SAM) model for medical images, provide training pipelines, inference scripts, GUI, and comparison benchmarks.

##### File structure
- Top-level directories and files:
- segment_anything/ – the core SAM model implementation (encapsulates model architecture, prompt encoder, mask decoder, utilities).
- extensions/ – optional prompt-based segmentation extensions (point, text, sparse-marker).
- comparisons/ – inference and training scripts to benchmark against other methods (DeepLabV3+, nnU-Net, vanilla SAM).
- utils/ – shared utilities for preprocessing, checkpoint conversion, evaluation metrics (SurfaceDice), data formatting.
- train_one_gpu.py, train_multi_gpus.py & train_multi_gpus.sh – training entrypoints and SLURM script for distributed training.
- pre_CT_MR.py – data preprocessing: windowing, normalization, resampling.
- MedSAM_Inference.py, gui.py – inference entrypoints: CLI and PyQt5 GUI.
- work_dir/ – recommended filesystem layout for checkpoints and intermediate data.

##### README for additional insights
- Architecture: based on Meta’s SAM (Vision Transformer + prompt encoder + mask decoder).
- Dependencies: PyTorch 2.0, cc3d, PyQt5.
- Workflows: data preprocessing → training/fine-tuning → ckpt conversion → inference (CLI, notebook, GUI).

##### Detail
- Subcomponents within segment_anything (prompt encoder, transformer, mask decoder).
- Optional flows through extensions (e.g., point_prompt/train_point_prompt.py).
- Both single-GPU and distributed-GPU training paths.
- Don’t overload: group similar scripts under “Training Pipeline” but call out the SLURM script separately.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RAW["Raw DICOM/CT/MR"]:::data_storage
        PREPROCESS["pre_CT_MR.py"]:::process_blue
        NPY["NPY Dataset"]:::data_storage
        RAW -->|"windowing,resampling"| PREPROCESS
        PREPROCESS -->|"output NPY"| NPY
    end

    %% Shared Utils Preprocessing wrapper
    UTIL_PRE["utils/pre_CT_MR.py"]:::process_blue
    NPY --> UTIL_PRE
    click PREPROCESS "https://github.com/bowang-lab/medsam/blob/main/pre_CT_MR.py"
    click UTIL_PRE "https://github.com/bowang-lab/medsam/blob/main/utils/pre_CT_MR.py"

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        TRAIN1["train_one_gpu.py"]:::process_green
        TRAINM["train_multi_gpus.py"]:::process_green
        SLURM["train_multi_gpus.sh"]:::process_green
        CHECKPTS["Model Checkpoints"]:::data_storage
        NPY --> TRAIN1
        NPY --> TRAINM
        TRAINM --> SLURM
        TRAIN1 --> CHECKPTS
        SLURM --> CHECKPTS
    end
    click TRAIN1 "https://github.com/bowang-lab/medsam/blob/main/train_one_gpu.py"
    click TRAINM "https://github.com/bowang-lab/medsam/blob/main/train_multi_gpus.py"
    click SLURM "https://github.com/bowang-lab/medsam/blob/main/train_multi_gpus.sh"

    %% Checkpoint Conversion
    CONVERT["utils/ckpt_convert.py"]:::process_blue
    CHECKPTS --> CONVERT
    click CONVERT "https://github.com/bowang-lab/medsam/blob/main/utils/ckpt_convert.py"

    %% Core Model Library
    subgraph "SAM Core Library" 
        direction TB
        ROOT["segment_anything/"]:::library_orange
        BUILD["build_sam.py"]:::library_orange
        SAMMOD["modeling/sam.py"]:::library_orange
        IMGENC["modeling/image_encoder.py"]:::library_orange
        PROMENC["modeling/prompt_encoder.py"]:::library_orange
        MASKDEC["modeling/mask_decoder.py"]:::library_orange
        TRANS["modeling/transformer.py"]:::library_orange
        COMMON["modeling/common.py"]:::library_orange
        PRED["predictor.py"]:::library_orange
        ONNX["utils/onnx.py"]:::library_orange
        TRANSF["utils/transforms.py"]:::library_orange
        AUTOAMG["automatic_mask_generator.py"]:::library_orange

        ROOT --> BUILD
        ROOT --> SAMMOD
        SAMMOD --> IMGENC
        SAMMOD --> PROMENC
        SAMMOD --> MASKDEC
        SAMMOD --> TRANS
        SAMMOD --> COMMON
        ROOT --> PRED
        ROOT --> ONNX
        ROOT --> TRANSF
        ROOT --> AUTOAMG
    end
    click ROOT "https://github.com/bowang-lab/medsam/tree/main/segment_anything/"
    click IMGENC "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/image_encoder.py"
    click PROMENC "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/prompt_encoder.py"
    click MASKDEC "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/mask_decoder.py"
    click TRANS "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/transformer.py"
    click COMMON "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/common.py"
    click SAMMOD "https://github.com/bowang-lab/medsam/blob/main/segment_anything/modeling/sam.py"
    click BUILD "https://github.com/bowang-lab/medsam/blob/main/segment_anything/build_sam.py"
    click PRED "https://github.com/bowang-lab/medsam/blob/main/segment_anything/predictor.py"
    click ONNX "https://github.com/bowang-lab/medsam/blob/main/segment_anything/utils/onnx.py"
    click TRANSF "https://github.com/bowang-lab/medsam/blob/main/segment_anything/utils/transforms.py"
    click AUTOAMG "https://github.com/bowang-lab/medsam/blob/main/segment_anything/automatic_mask_generator.py"

    %% Inference Interfaces
    subgraph "Inference Interfaces" 
        direction TB
        CLI["MedSAM_Inference.py"]:::process_purple
        GUI["gui.py"]:::process_purple
        NOTE["tutorial_quickstart.ipynb"]:::process_purple
        CONVERT --> ROOT
        NPY --> ROOT
        ROOT --> CLI
        ROOT --> GUI
        ROOT --> NOTE
        CLI --> SEG_OUT
        GUI --> SEG_OUT
        NOTE --> SEG_OUT
    end
    click CLI "https://github.com/bowang-lab/medsam/blob/main/MedSAM_Inference.py"
    click GUI "https://github.com/bowang-lab/medsam/blob/main/gui.py"
    click NOTE "https://github.com/bowang-lab/medsam/blob/main/tutorial_quickstart.ipynb"

    SEG_OUT["Segmentation Masks"]:::data_storage

    %% Extensions Layer
    subgraph "Extensions"
        direction TB
        subgraph "Point Prompt" 
            direction TB
            EP1["extensions/point_prompt/train_point_prompt.py"]:::process_gray
            ET1["extensions/point_prompt/tutorial_point_prompt_seg.ipynb"]:::process_gray
        end
        subgraph "Text Prompt"
            direction TB
            EP2["extensions/text_prompt/train_text_prompt.py"]:::process_gray
            ET2["extensions/text_prompt/tutorial_text_prompt_seg.ipynb"]:::process_gray
        end
        subgraph "3D Sparse Marker"
            direction TB
            EM1["extensions/seg_3dnii_sparse_marker/label_interpolate.py"]:::process_gray
            EM2["extensions/seg_3dnii_sparse_marker/medsam_infer_3Dbox_adrenal.py"]:::process_gray
        end
        NPY --> EP1
        NPY --> EP2
        NPY --> EM1
    end
    click EP1 "https://github.com/bowang-lab/medsam/blob/main/extensions/point_prompt/train_point_prompt.py"
    click ET1 "https://github.com/bowang-lab/medsam/blob/main/extensions/point_prompt/tutorial_point_prompt_seg.ipynb"
    click EP2 "https://github.com/bowang-lab/medsam/blob/main/extensions/text_prompt/train_text_prompt.py"
    click ET2 "https://github.com/bowang-lab/medsam/blob/main/extensions/text_prompt/tutorial_text_prompt_seg.ipynb"
    click EM1 "https://github.com/bowang-lab/medsam/blob/main/extensions/seg_3dnii_sparse_marker/label_interpolate.py"
    click EM2 "https://github.com/bowang-lab/medsam/blob/main/extensions/seg_3dnii_sparse_marker/medsam_infer_3Dbox_adrenal.py"

    %% Benchmarking Layer
    subgraph "Benchmarking"
        direction TB
        B1["comparisons/DeepLabV3+ scripts"]:::process_gray
        B2["comparisons/SAM scripts"]:::process_gray
        B3["comparisons/nnU-Net scripts"]:::process_gray
        NPY --> B1
        NPY --> B2
        NPY --> B3
        B1 --> EVAL
        B2 --> EVAL
        B3 --> EVAL
    end
    click B1 "https://github.com/bowang-lab/medsam/tree/main/comparisons/DeepLabV3+/*"
    click B2 "https://github.com/bowang-lab/medsam/tree/main/comparisons/SAM/*"
    click B3 "https://github.com/bowang-lab/medsam/tree/main/comparisons/nnU-Net/*"

    EVAL["Evaluation Metrics (SurfaceDice)"]:::process_blue
    click EVAL "https://github.com/bowang-lab/medsam/blob/main/utils/SurfaceDice.py"

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PT["PyTorch 2.0"]:::external
        CC3D["cc3d"]:::external
        QT["PyQt5"]:::external
    end
    PT --> TRAIN1
    PT --> TRAINM
    PT --> ROOT
    PT --> CLI
    CC3D --> AUTOAMG
    QT --> GUI

    %% Styles
    classDef data_storage fill:#CEF6F5,stroke:#028482,color:#000
    classDef process_blue fill:#D0E1F9,stroke:#3A70A1,color:#000
    classDef process_green fill:#D5F5E3,stroke:#28B463,color:#000
    classDef library_orange fill:#FDEBD0,stroke:#CA6F1E,color:#000
    classDef process_purple fill:#F4ECF7,stroke:#884EA0,color:#000
    classDef process_gray fill:#E5E7E9,stroke:#7B7D7D,color:#000
    classDef external fill:#FDEDEC,stroke:#C0392B,color:#000
```
