How facebookresearch/sam3 works
SAM 3 is a promptable image/video segmentation foundation-model package, with training, evaluation, and an LLM-assisted agent as adjacent capabilities. The principal workflow builds a model, accepts text or visual prompts, and returns masks, boxes, and scores; video prediction adds session-based prompts and propagation. The graph combines documented workflows with relationships visible in the supplied source index. Several components are unsampled, so their internal calls are not inferred.
Components 25
Inference APIs
- Model builders
sam3/model_builder.py - Image processor
sam3/model/sam3_image_processor.py - Video predictor
sam3/model/sam3_video_predictor.py - Video inference
sam3/model/sam3_video_inference.py
Model Architecture
- Image model
sam3/model/sam3_image.py - Video tracking
sam3/model/sam3_video_base.py - Multiplex tracking
sam3/model/video_tracking_multiplex.py - Mask decoder
sam3/sam/mask_decoder.py - SAM transformer
sam3/sam/transformer.py - Tracking memory
sam3/model/memory.py - Geometry encoding
sam3/model/geometry_encoders.py - Tensor structures
sam3/model/data_misc.py - Box operations
sam3/model/box_ops.py - Model utilities
sam3/model/model_misc.py
Agent
- Agent inference
sam3/agent/inference.py - Agent orchestration
sam3/agent/agent_core.py - LLM client
sam3/agent/client_llm.py - SAM client
sam3/agent/client_sam3.py
Training
- Trainer
sam3/train/trainer.py - Training datasets
sam3/train/data - Training losses
sam3/train/loss/loss_fns.py
Evaluation
- Evaluation suite
sam3/eval - SA-Co video evaluation
sam3/eval/saco_veval_evaluators.py
Other components
- User
- Evaluator
Connections 28
- User → Model builders: requests model
- Model builders → Image model: builds image model
sam3/model_builder.py - Model builders → Video predictor: builds predictor
sam3/model_builder.py - User → Image processor: sets image prompt
- Image processor → Image model: runs inference
- Image processor → User: returns masks boxes scores
- User → Video predictor: starts session
- Video predictor → Video inference: handles requests
sam3/model_builder.py - Video inference → Video tracking: propagates frames
sam3/model/sam3_video_inference.py - Video predictor → User: streams outputs
- Model builders → Mask decoder: configures decoder
sam3/model_builder.py - Model builders → Geometry encoding: configures geometry
sam3/model_builder.py - Video tracking → Tracking memory: uses memory
- Multiplex tracking → Tracking memory: reads memory
sam3/model/video_tracking_multiplex.py - Multiplex tracking → Tensor structures: uses structures
sam3/model/video_tracking_multiplex.py - Multiplex tracking → Mask decoder: uses decoder
sam3/model/video_tracking_multiplex.py - Multiplex tracking → SAM transformer: uses transformer
sam3/model/video_tracking_multiplex.py - Geometry encoding → Box operations: uses box operations
sam3/model/geometry_encoders.py - Geometry encoding → Model utilities: uses utilities
sam3/model/geometry_encoders.py - User → Agent inference: requests segmentation
- Agent inference → Agent orchestration: invokes agent
sam3/agent/inference.py - Agent orchestration → LLM client: uses language client
- Agent orchestration → SAM client: uses segmentation client
- Trainer → Training datasets: uses datasets
sam3/train/trainer.py - Trainer → Training losses: computes loss
- Evaluator → Evaluation suite: runs evaluation
- Evaluator → SA-Co video evaluation: evaluates video
- SA-Co video evaluation → Evaluation suite: uses metric tooling
sam3/eval/saco_veval_evaluators.py