# supermedintel/medical-sam2 architecture

> Architecture diagram of the GitHub repository supermedintel/medical-sam2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/supermedintel/medical-sam2
- Repository: https://github.com/supermedintel/medical-sam2

## Overview

##### Main Components Identification

The system should be divided into these major components:

a) Core Model Architecture
- SAM2 Base Model
- 2D Processing Pipeline
- 3D Processing Pipeline
- Memory and Attention Mechanisms

b) Data Processing Components
- 2D Function Module
- 3D Function Module
- Dataset Handlers

**Input Layer -> Processing Pipeline -> Model Core -> Output Layer**

##### Key Relationships to Highlight

- How the 2D and 3D pipelines share the same core SAM2 architecture
- Memory attention mechanism's role in processing sequential data
- Demonstrate how the automatic mask generator interfaces with the core model
- Configuration flow from yaml files to model components

##### Technical Details

- Input image sizes (1024x1024 as specified in the training scripts)
- Model variant indicators (SAM2 Hierarchical Small/Tiny)
- Dataset-specific processing paths
- Checkpoint loading mechanism

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        I1[Input Images]
    end

    subgraph "Data Processing Layer"
        D2D["2D Dataset Handler"]:::data
        D3D["3D Dataset Handler"]:::data
        AMOS["AMOS Dataset"]:::data
        BTCV["BTCV Dataset"]:::data
        P2D["2D Processing Pipeline"]:::process
        P3D["3D Processing Pipeline"]:::process
    end

    subgraph "Core Model Layer"
        SAM["SAM2 Base Model"]:::core
        subgraph "Encoders"
            IE["Image Encoder"]:::core
            PE["Prompt Encoder"]:::core
        end
        MAS["Memory Attention System"]:::core
        POS["Position Encoding"]:::core
        MD["Mask Decoder"]:::core
    end

    subgraph "Training System"
        T2D["2D Training"]:::train
        T3D["3D Training"]:::train
    end

    subgraph "Inference System"
        IP["Image Predictor"]:::infer
        VP["Video Predictor"]:::infer
    end

    subgraph "Configuration"
        CONF["Global Settings"]:::config
        MC["Model Configs"]:::config
        CUDA["CUDA Operations"]:::config
    end

    I1 --> D2D & D3D
    D3D --> AMOS & BTCV
    D2D --> P2D
    D3D --> P3D
    P2D & P3D --> SAM
    SAM --> IE & PE
    IE & PE --> MAS
    MAS --> MD
    POS --> MAS
    MD --> IP & VP
    CONF --> SAM
    MC --> SAM
    CUDA --> MD

    %% Click Events
    click SAM "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/sam2_base.py"
    click IE "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/backbones/image_encoder.py"
    click PE "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/sam/prompt_encoder.py"
    click MD "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/sam/mask_decoder.py"
    click MAS "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/memory_attention.py"
    click POS "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/modeling/position_encoding.py"
    click P2D "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/func_2d/function.py"
    click P3D "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/func_3d/function.py"
    click D2D "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/func_2d/dataset.py"
    click AMOS "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/func_3d/dataset/amos.py"
    click BTCV "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/func_3d/dataset/btcv.py"
    click T2D "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/train_2d.py"
    click T3D "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/train_3d.py"
    click IP "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/sam2_image_predictor.py"
    click VP "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/sam2_video_predictor.py"
    click CONF "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/conf/global_settings.py"
    click MC "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/sam2_hiera_s.yaml"
    click CUDA "https://github.com/SuperMedIntel/Medical-SAM2/blob/main/sam2_train/csrc/connected_components.cu"

    %% Styles
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef data fill:#2ecc71,stroke:#27ae60,color:white
    classDef process fill:#e67e22,stroke:#d35400,color:white
    classDef train fill:#9b59b6,stroke:#8e44ad,color:white
    classDef infer fill:#e74c3c,stroke:#c0392b,color:white
    classDef config fill:#34495e,stroke:#2c3e50,color:white

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[Data Handlers]:::data
        L3[Processing]:::process
        L4[Training]:::train
        L5[Inference]:::infer
        L6[Configuration]:::config
    end
```
