# facebookresearch/sam2 architecture

> Architecture diagram of the GitHub repository facebookresearch/sam2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/sam2
- Repository: https://github.com/facebookresearch/sam2

## Overview

SAM 2 provides promptable segmentation for images and videos, with video mask propagation through streaming memory. The user workflow builds a checkpoint-backed model, accepts image or video prompts, and returns masks; automatic image masks are a separate capability. The repository also includes video-model training and SA-V evaluation. The graph follows documented entry points and indexed module references; some internal model wiring is configuration-driven or unsampled, so those relationships are omitted or marked unverified.

Main entry points: [`sam2/modeling/sam2_base.py`](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam2_base.py) (SAM2 Core), [`sam2/sam2_video_predictor.py`](https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_video_predictor.py) (Video Predictor), [`sam2/sam2_image_predictor.py`](https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_image_predictor.py) (Image Predictor), [`sam2/build_sam.py`](https://github.com/facebookresearch/sam2/blob/main/sam2/build_sam.py) (Model Builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Prediction APIs"]
  node_builder["Model Builder<br/>[build_sam.py]"]
  node_image_api["Image Predictor"]
  node_video_api["Video Predictor"]
  node_auto_masks["Automatic Masks"]
  node_video_frames["Video Loading<br/>[misc.py]"]
  node_vos_tool["VOS Inference<br/>[vos_inference.py]"]
end

subgraph group_model["Segmentation Model"]
  node_base["SAM2 Core<br/>[sam2_base.py]"]
  node_backbone["Image Backbone<br/>[hieradet.py]"]
  node_image_encoder["Feature Encoder<br/>[image_encoder.py]"]
  node_prompt_encoder["Prompt Encoder<br/>[prompt_encoder.py]"]
  node_mask_decoder["Mask Decoder<br/>[mask_decoder.py]"]
  node_memory_attention["Memory Attention"]
  node_memory_encoder["Memory Encoder<br/>[memory_encoder.py]"]
  node_position_encoding["Position Encoding"]
end

subgraph group_training["Training"]
  node_train_data["VOS Dataset<br/>[vos_dataset.py]"]
  node_segment_loader["Segment Loaders"]
  node_train_model["Training Model<br/>[sam2.py]"]
  node_trainer["Trainer<br/>[trainer.py]"]
  node_optimizer["Optimizer<br/>[optimizer.py]"]
  node_train_data_utils["Batch Data<br/>[data_utils.py]"]
end

subgraph group_evaluation["Dataset Evaluation"]
  node_sav_eval["SA-V Evaluator<br/>[sav_evaluator.py]"]
  node_benchmark["Video Metrics<br/>[sav_benchmark.py]"]
end

node_user(("User"))
node_predictions["Predicted Masks"]

node_user -->|"selects checkpoint"| node_builder
node_builder -->|"builds model for"| node_image_api
node_builder -->|"builds predictor"| node_video_api
node_user -->|"prompts image"| node_image_api
node_image_api -->|"predicts masks"| node_base
node_auto_masks -->|"uses predictor"| node_image_api
node_user -->|"adds prompts"| node_video_api
node_video_api -->|"loads frames"| node_video_frames
node_video_api -->|"tracks frames"| node_base
node_base -->|"encodes memory"| node_memory_encoder
node_base -->|"uses decoder"| node_mask_decoder
node_base -->|"uses prompts"| node_prompt_encoder
node_base -.->|"extracts features"| node_backbone
node_backbone -.->|"provides features"| node_image_encoder
node_base -.->|"attends to memory"| node_memory_attention
node_prompt_encoder -->|"uses encoding"| node_position_encoding
node_mask_decoder -->|"uses encoding"| node_position_encoding
node_video_api -->|"returns masks"| node_predictions
node_image_api -->|"returns masks"| node_predictions
node_user -->|"runs inference"| node_vos_tool
node_vos_tool -->|"builds predictor"| node_builder
node_train_data -->|"loads segments"| node_segment_loader
node_train_data -->|"uses batch data"| node_train_data_utils
node_trainer -->|"configures optimizer"| node_optimizer
node_trainer -.->|"trains model"| node_train_model
node_train_model -->|"extends core"| node_base
node_sav_eval -->|"evaluates masks"| node_benchmark

click node_builder "https://github.com/facebookresearch/sam2/blob/main/sam2/build_sam.py"
click node_image_api "https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_image_predictor.py"
click node_video_api "https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_video_predictor.py"
click node_auto_masks "https://github.com/facebookresearch/sam2/blob/main/sam2/automatic_mask_generator.py"
click node_video_frames "https://github.com/facebookresearch/sam2/blob/main/sam2/utils/misc.py"
click node_base "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam2_base.py"
click node_backbone "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/backbones/hieradet.py"
click node_image_encoder "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/backbones/image_encoder.py"
click node_prompt_encoder "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam/prompt_encoder.py"
click node_mask_decoder "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam/mask_decoder.py"
click node_memory_attention "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/memory_attention.py"
click node_memory_encoder "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/memory_encoder.py"
click node_position_encoding "https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/position_encoding.py"
click node_vos_tool "https://github.com/facebookresearch/sam2/blob/main/tools/vos_inference.py"
click node_train_data "https://github.com/facebookresearch/sam2/blob/main/training/dataset/vos_dataset.py"
click node_segment_loader "https://github.com/facebookresearch/sam2/blob/main/training/dataset/vos_segment_loader.py"
click node_train_model "https://github.com/facebookresearch/sam2/blob/main/training/model/sam2.py"
click node_trainer "https://github.com/facebookresearch/sam2/blob/main/training/trainer.py"
click node_optimizer "https://github.com/facebookresearch/sam2/blob/main/training/optimizer.py"
click node_train_data_utils "https://github.com/facebookresearch/sam2/blob/main/training/utils/data_utils.py"
click node_sav_eval "https://github.com/facebookresearch/sam2/blob/main/sav_dataset/sav_evaluator.py"
click node_benchmark "https://github.com/facebookresearch/sam2/blob/main/sav_dataset/utils/sav_benchmark.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_builder,node_image_api,node_video_api,node_auto_masks,node_video_frames,node_vos_tool,node_user toneBlue
class node_base,node_backbone,node_image_encoder,node_prompt_encoder,node_mask_decoder,node_memory_attention,node_memory_encoder,node_position_encoding toneAmber
class node_train_data,node_segment_loader,node_train_model,node_trainer,node_optimizer,node_train_data_utils toneMint
class node_sav_eval,node_benchmark toneRose
class node_predictions toneIndigo
```

## Components

### Prediction APIs

- [Model Builder](https://github.com/facebookresearch/sam2/blob/main/sam2/build_sam.py): `sam2/build_sam.py`
- [Image Predictor](https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_image_predictor.py): `sam2/sam2_image_predictor.py`
- [Video Predictor](https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_video_predictor.py): `sam2/sam2_video_predictor.py`
- [Automatic Masks](https://github.com/facebookresearch/sam2/blob/main/sam2/automatic_mask_generator.py): `sam2/automatic_mask_generator.py`
- [Video Loading](https://github.com/facebookresearch/sam2/blob/main/sam2/utils/misc.py): `sam2/utils/misc.py`
- [VOS Inference](https://github.com/facebookresearch/sam2/blob/main/tools/vos_inference.py): `tools/vos_inference.py`

### Segmentation Model

- [SAM2 Core](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam2_base.py): `sam2/modeling/sam2_base.py`
- [Image Backbone](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/backbones/hieradet.py): `sam2/modeling/backbones/hieradet.py`
- [Feature Encoder](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/backbones/image_encoder.py): `sam2/modeling/backbones/image_encoder.py`
- [Prompt Encoder](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam/prompt_encoder.py): `sam2/modeling/sam/prompt_encoder.py`
- [Mask Decoder](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/sam/mask_decoder.py): `sam2/modeling/sam/mask_decoder.py`
- [Memory Attention](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/memory_attention.py): `sam2/modeling/memory_attention.py`
- [Memory Encoder](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/memory_encoder.py): `sam2/modeling/memory_encoder.py`
- [Position Encoding](https://github.com/facebookresearch/sam2/blob/main/sam2/modeling/position_encoding.py): `sam2/modeling/position_encoding.py`

### Training

- [VOS Dataset](https://github.com/facebookresearch/sam2/blob/main/training/dataset/vos_dataset.py): `training/dataset/vos_dataset.py`
- [Segment Loaders](https://github.com/facebookresearch/sam2/blob/main/training/dataset/vos_segment_loader.py): `training/dataset/vos_segment_loader.py`
- [Training Model](https://github.com/facebookresearch/sam2/blob/main/training/model/sam2.py): `training/model/sam2.py`
- [Trainer](https://github.com/facebookresearch/sam2/blob/main/training/trainer.py): `training/trainer.py`
- [Optimizer](https://github.com/facebookresearch/sam2/blob/main/training/optimizer.py): `training/optimizer.py`
- [Batch Data](https://github.com/facebookresearch/sam2/blob/main/training/utils/data_utils.py): `training/utils/data_utils.py`

### Dataset Evaluation

- [SA-V Evaluator](https://github.com/facebookresearch/sam2/blob/main/sav_dataset/sav_evaluator.py): `sav_dataset/sav_evaluator.py`
- [Video Metrics](https://github.com/facebookresearch/sam2/blob/main/sav_dataset/utils/sav_benchmark.py): `sav_dataset/utils/sav_benchmark.py`

### Other components

- **User**
- **Predicted Masks**

## Connections

- User → Model Builder: selects checkpoint (evidence: `README.md`)
- Model Builder → Image Predictor: builds model for (evidence: `README.md`)
- Model Builder → Video Predictor: builds predictor (evidence: `README.md`)
- User → Image Predictor: prompts image (evidence: `README.md`)
- Image Predictor → SAM2 Core: predicts masks (evidence: `sam2/sam2_image_predictor.py`)
- Automatic Masks → Image Predictor: uses predictor (evidence: `sam2/automatic_mask_generator.py`)
- User → Video Predictor: adds prompts (evidence: `README.md`)
- Video Predictor → Video Loading: loads frames (evidence: `sam2/utils/misc.py`)
- Video Predictor → SAM2 Core: tracks frames (evidence: `sam2/sam2_video_predictor.py`)
- SAM2 Core → Memory Encoder: encodes memory
- SAM2 Core → Mask Decoder: uses decoder (evidence: `sam2/modeling/sam2_base.py`)
- SAM2 Core → Prompt Encoder: uses prompts (evidence: `sam2/modeling/sam2_base.py`)
- SAM2 Core → Image Backbone: extracts features
- Image Backbone → Feature Encoder: provides features
- SAM2 Core → Memory Attention: attends to memory
- Prompt Encoder → Position Encoding: uses encoding (evidence: `sam2/modeling/sam/prompt_encoder.py`)
- Mask Decoder → Position Encoding: uses encoding (evidence: `sam2/modeling/sam/mask_decoder.py`)
- Video Predictor → Predicted Masks: returns masks (evidence: `README.md`)
- Image Predictor → Predicted Masks: returns masks (evidence: `README.md`)
- User → VOS Inference: runs inference (evidence: `tools/vos_inference.py`)
- VOS Inference → Model Builder: builds predictor (evidence: `tools/vos_inference.py`)
- VOS Dataset → Segment Loaders: loads segments (evidence: `training/dataset/vos_dataset.py`)
- VOS Dataset → Batch Data: uses batch data (evidence: `training/dataset/vos_dataset.py`)
- Trainer → Optimizer: configures optimizer (evidence: `training/trainer.py`)
- Trainer → Training Model: trains model
- Training Model → SAM2 Core: extends core (evidence: `training/model/sam2.py`)
- SA-V Evaluator → Video Metrics: evaluates masks (evidence: `sav_dataset/sav_evaluator.py`)
