How facebookresearch/sam2 works
SAM 2 provides promptable segmentation for images and videos, with video mask propagation through streaming memory. The user workflow builds a checkpoint-backed model, accepts image or video prompts, and returns masks; automatic image masks are a separate capability. The repository also includes video-model training and SA-V evaluation. The graph follows documented entry points and indexed module references; some internal model wiring is configuration-driven or unsampled, so those relationships are omitted or marked unverified.
Main entry points: sam2_base.py SAM2 Core · sam2_video_predictor.py Video Predictor · sam2_image_predictor.py Image Predictor · build_sam.py Model Builder
Components 24
Prediction APIs
- Model Builder
sam2/build_sam.py - Image Predictor
sam2/sam2_image_predictor.py - Video Predictor
sam2/sam2_video_predictor.py - Automatic Masks
sam2/automatic_mask_generator.py - Video Loading
sam2/utils/misc.py - VOS Inference
tools/vos_inference.py
Segmentation Model
- SAM2 Core
sam2/modeling/sam2_base.py - Image Backbone
sam2/modeling/backbones/hieradet.py - Feature Encoder
sam2/modeling/backbones/image_encoder.py - Prompt Encoder
sam2/modeling/sam/prompt_encoder.py - Mask Decoder
sam2/modeling/sam/mask_decoder.py - Memory Attention
sam2/modeling/memory_attention.py - Memory Encoder
sam2/modeling/memory_encoder.py - Position Encoding
sam2/modeling/position_encoding.py
Training
- VOS Dataset
training/dataset/vos_dataset.py - Segment Loaders
training/dataset/vos_segment_loader.py - Training Model
training/model/sam2.py - Trainer
training/trainer.py - Optimizer
training/optimizer.py - Batch Data
training/utils/data_utils.py
Dataset Evaluation
- SA-V Evaluator
sav_dataset/sav_evaluator.py - Video Metrics
sav_dataset/utils/sav_benchmark.py
Other components
- User
- Predicted Masks
Connections 27
- User → Model Builder: selects checkpoint
README.md - Model Builder → Image Predictor: builds model for
README.md - Model Builder → Video Predictor: builds predictor
README.md - User → Image Predictor: prompts image
README.md - Image Predictor → SAM2 Core: predicts masks
sam2/sam2_image_predictor.py - Automatic Masks → Image Predictor: uses predictor
sam2/automatic_mask_generator.py - User → Video Predictor: adds prompts
README.md - Video Predictor → Video Loading: loads frames
sam2/utils/misc.py - Video Predictor → SAM2 Core: tracks frames
sam2/sam2_video_predictor.py - SAM2 Core → Memory Encoder: encodes memory
- SAM2 Core → Mask Decoder: uses decoder
sam2/modeling/sam2_base.py - SAM2 Core → Prompt Encoder: uses prompts
sam2/modeling/sam2_base.py - SAM2 Core → Image Backbone: extracts features
- Image Backbone → Feature Encoder: provides features
- SAM2 Core → Memory Attention: attends to memory
- Prompt Encoder → Position Encoding: uses encoding
sam2/modeling/sam/prompt_encoder.py - Mask Decoder → Position Encoding: uses encoding
sam2/modeling/sam/mask_decoder.py - Video Predictor → Predicted Masks: returns masks
README.md - Image Predictor → Predicted Masks: returns masks
README.md - User → VOS Inference: runs inference
tools/vos_inference.py - VOS Inference → Model Builder: builds predictor
tools/vos_inference.py - VOS Dataset → Segment Loaders: loads segments
training/dataset/vos_dataset.py - VOS Dataset → Batch Data: uses batch data
training/dataset/vos_dataset.py - Trainer → Optimizer: configures optimizer
training/trainer.py - Trainer → Training Model: trains model
- Training Model → SAM2 Core: extends core
training/model/sam2.py - SA-V Evaluator → Video Metrics: evaluates masks
sav_dataset/sav_evaluator.py