# facebookresearch/sam3 architecture

> Architecture diagram of the GitHub repository facebookresearch/sam3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/sam3
- Repository: https://github.com/facebookresearch/sam3
- Video tour (about a minute): https://gitdiagram.com/facebookresearch/sam3/video

## Overview

SAM 3 is a promptable image/video segmentation foundation-model package, with training, evaluation, and an LLM-assisted agent as adjacent capabilities. The principal workflow builds a model, accepts text or visual prompts, and returns masks, boxes, and scores; video prediction adds session-based prompts and propagation. The graph combines documented workflows with relationships visible in the supplied source index. Several components are unsampled, so their internal calls are not inferred.

Main entry points: [`sam3/model_builder.py`](https://github.com/facebookresearch/sam3/blob/main/sam3/model_builder.py) (Model builders), [`sam3/model/video_tracking_multiplex.py`](https://github.com/facebookresearch/sam3/blob/main/sam3/model/video_tracking_multiplex.py) (Multiplex tracking), [`sam3/model/sam3_video_predictor.py`](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_predictor.py) (Video predictor), [`sam3/model/sam3_image_processor.py`](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_image_processor.py) (Image processor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference APIs"]
  node_builder["Model builders<br/>[model_builder.py]"]
  node_image_api["Image processor"]
  node_video_api["Video predictor"]
  node_video_runtime["Video inference"]
end

subgraph group_architecture["Model Architecture"]
  node_image_model["Image model<br/>[sam3_image.py]"]
  node_video_model["Video tracking<br/>[sam3_video_base.py]"]
  node_multiplex["Multiplex tracking"]
  node_decoder["Mask decoder<br/>[mask_decoder.py]"]
  node_transformer["SAM transformer<br/>[transformer.py]"]
  node_memory["Tracking memory<br/>[memory.py]"]
  node_geometry["Geometry encoding"]
  node_tensor_data["Tensor structures<br/>[data_misc.py]"]
  node_box_ops["Box operations<br/>[box_ops.py]"]
  node_model_utils["Model utilities<br/>[model_misc.py]"]
end

subgraph group_agent["Agent"]
  node_agent_entry["Agent inference<br/>[inference.py]"]
  node_agent_core["Agent orchestration<br/>[agent_core.py]"]
  node_llm["LLM client<br/>[client_llm.py]"]
  node_sam_client["SAM client<br/>[client_sam3.py]"]
end

subgraph group_learning["Training"]
  node_trainer["Trainer<br/>[trainer.py]"]
  node_train_data["Training datasets"]
  node_train_loss["Training losses<br/>[loss_fns.py]"]
end

subgraph group_assessment["Evaluation"]
  node_eval_tools["Evaluation suite"]
  node_video_eval["SA-Co video evaluation"]
end

node_user(("User"))
node_eval_user(("Evaluator"))

node_user -->|"requests model"| node_builder
node_builder -->|"builds image model"| node_image_model
node_builder -->|"builds predictor"| node_video_api
node_user -->|"sets image prompt"| node_image_api
node_image_api -->|"runs inference"| node_image_model
node_image_api -->|"returns masks boxes scores"| node_user
node_user -->|"starts session"| node_video_api
node_video_api -->|"handles requests"| node_video_runtime
node_video_runtime -.->|"propagates frames"| node_video_model
node_video_api -->|"streams outputs"| node_user
node_builder -->|"configures decoder"| node_decoder
node_builder -->|"configures geometry"| node_geometry
node_video_model -->|"uses memory"| node_memory
node_multiplex -->|"reads memory"| node_memory
node_multiplex -->|"uses structures"| node_tensor_data
node_multiplex -->|"uses decoder"| node_decoder
node_multiplex -->|"uses transformer"| node_transformer
node_geometry -->|"uses box operations"| node_box_ops
node_geometry -->|"uses utilities"| node_model_utils
node_user -.->|"requests segmentation"| node_agent_entry
node_agent_entry -->|"invokes agent"| node_agent_core
node_agent_core -.->|"uses language client"| node_llm
node_agent_core -.->|"uses segmentation client"| node_sam_client
node_trainer -->|"uses datasets"| node_train_data
node_trainer -.->|"computes loss"| node_train_loss
node_eval_user -.->|"runs evaluation"| node_eval_tools
node_eval_user -.->|"evaluates video"| node_video_eval
node_video_eval -->|"uses metric tooling"| node_eval_tools

click node_builder "https://github.com/facebookresearch/sam3/blob/main/sam3/model_builder.py"
click node_image_api "https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_image_processor.py"
click node_video_api "https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_predictor.py"
click node_video_runtime "https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_inference.py"
click node_image_model "https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_image.py"
click node_video_model "https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_base.py"
click node_multiplex "https://github.com/facebookresearch/sam3/blob/main/sam3/model/video_tracking_multiplex.py"
click node_decoder "https://github.com/facebookresearch/sam3/blob/main/sam3/sam/mask_decoder.py"
click node_transformer "https://github.com/facebookresearch/sam3/blob/main/sam3/sam/transformer.py"
click node_memory "https://github.com/facebookresearch/sam3/blob/main/sam3/model/memory.py"
click node_geometry "https://github.com/facebookresearch/sam3/blob/main/sam3/model/geometry_encoders.py"
click node_tensor_data "https://github.com/facebookresearch/sam3/blob/main/sam3/model/data_misc.py"
click node_box_ops "https://github.com/facebookresearch/sam3/blob/main/sam3/model/box_ops.py"
click node_model_utils "https://github.com/facebookresearch/sam3/blob/main/sam3/model/model_misc.py"
click node_agent_entry "https://github.com/facebookresearch/sam3/blob/main/sam3/agent/inference.py"
click node_agent_core "https://github.com/facebookresearch/sam3/blob/main/sam3/agent/agent_core.py"
click node_llm "https://github.com/facebookresearch/sam3/blob/main/sam3/agent/client_llm.py"
click node_sam_client "https://github.com/facebookresearch/sam3/blob/main/sam3/agent/client_sam3.py"
click node_trainer "https://github.com/facebookresearch/sam3/blob/main/sam3/train/trainer.py"
click node_train_data "https://github.com/facebookresearch/sam3/tree/main/sam3/train/data"
click node_train_loss "https://github.com/facebookresearch/sam3/blob/main/sam3/train/loss/loss_fns.py"
click node_eval_tools "https://github.com/facebookresearch/sam3/tree/main/sam3/eval"
click node_video_eval "https://github.com/facebookresearch/sam3/blob/main/sam3/eval/saco_veval_evaluators.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_builder,node_image_api,node_video_api,node_video_runtime,node_user toneBlue
class node_image_model,node_video_model,node_multiplex,node_decoder,node_transformer,node_memory,node_geometry,node_tensor_data,node_box_ops,node_model_utils toneAmber
class node_agent_entry,node_agent_core,node_llm,node_sam_client toneMint
class node_trainer,node_train_data,node_train_loss toneRose
class node_eval_tools,node_video_eval,node_eval_user toneIndigo
```

## Components

### Inference APIs

- [Model builders](https://github.com/facebookresearch/sam3/blob/main/sam3/model_builder.py): `sam3/model_builder.py`
- [Image processor](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_image_processor.py): `sam3/model/sam3_image_processor.py`
- [Video predictor](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_predictor.py): `sam3/model/sam3_video_predictor.py`
- [Video inference](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_inference.py): `sam3/model/sam3_video_inference.py`

### Model Architecture

- [Image model](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_image.py): `sam3/model/sam3_image.py`
- [Video tracking](https://github.com/facebookresearch/sam3/blob/main/sam3/model/sam3_video_base.py): `sam3/model/sam3_video_base.py`
- [Multiplex tracking](https://github.com/facebookresearch/sam3/blob/main/sam3/model/video_tracking_multiplex.py): `sam3/model/video_tracking_multiplex.py`
- [Mask decoder](https://github.com/facebookresearch/sam3/blob/main/sam3/sam/mask_decoder.py): `sam3/sam/mask_decoder.py`
- [SAM transformer](https://github.com/facebookresearch/sam3/blob/main/sam3/sam/transformer.py): `sam3/sam/transformer.py`
- [Tracking memory](https://github.com/facebookresearch/sam3/blob/main/sam3/model/memory.py): `sam3/model/memory.py`
- [Geometry encoding](https://github.com/facebookresearch/sam3/blob/main/sam3/model/geometry_encoders.py): `sam3/model/geometry_encoders.py`
- [Tensor structures](https://github.com/facebookresearch/sam3/blob/main/sam3/model/data_misc.py): `sam3/model/data_misc.py`
- [Box operations](https://github.com/facebookresearch/sam3/blob/main/sam3/model/box_ops.py): `sam3/model/box_ops.py`
- [Model utilities](https://github.com/facebookresearch/sam3/blob/main/sam3/model/model_misc.py): `sam3/model/model_misc.py`

### Agent

- [Agent inference](https://github.com/facebookresearch/sam3/blob/main/sam3/agent/inference.py): `sam3/agent/inference.py`
- [Agent orchestration](https://github.com/facebookresearch/sam3/blob/main/sam3/agent/agent_core.py): `sam3/agent/agent_core.py`
- [LLM client](https://github.com/facebookresearch/sam3/blob/main/sam3/agent/client_llm.py): `sam3/agent/client_llm.py`
- [SAM client](https://github.com/facebookresearch/sam3/blob/main/sam3/agent/client_sam3.py): `sam3/agent/client_sam3.py`

### Training

- [Trainer](https://github.com/facebookresearch/sam3/blob/main/sam3/train/trainer.py): `sam3/train/trainer.py`
- [Training datasets](https://github.com/facebookresearch/sam3/tree/main/sam3/train/data): `sam3/train/data`
- [Training losses](https://github.com/facebookresearch/sam3/blob/main/sam3/train/loss/loss_fns.py): `sam3/train/loss/loss_fns.py`

### Evaluation

- [Evaluation suite](https://github.com/facebookresearch/sam3/tree/main/sam3/eval): `sam3/eval`
- [SA-Co video evaluation](https://github.com/facebookresearch/sam3/blob/main/sam3/eval/saco_veval_evaluators.py): `sam3/eval/saco_veval_evaluators.py`

### Other components

- **User**
- **Evaluator**

## Connections

- User → Model builders: requests model
- Model builders → Image model: builds image model (evidence: `sam3/model_builder.py`)
- Model builders → Video predictor: builds predictor (evidence: `sam3/model_builder.py`)
- User → Image processor: sets image prompt
- Image processor → Image model: runs inference
- Image processor → User: returns masks boxes scores
- User → Video predictor: starts session
- Video predictor → Video inference: handles requests (evidence: `sam3/model_builder.py`)
- Video inference → Video tracking: propagates frames (evidence: `sam3/model/sam3_video_inference.py`)
- Video predictor → User: streams outputs
- Model builders → Mask decoder: configures decoder (evidence: `sam3/model_builder.py`)
- Model builders → Geometry encoding: configures geometry (evidence: `sam3/model_builder.py`)
- Video tracking → Tracking memory: uses memory
- Multiplex tracking → Tracking memory: reads memory (evidence: `sam3/model/video_tracking_multiplex.py`)
- Multiplex tracking → Tensor structures: uses structures (evidence: `sam3/model/video_tracking_multiplex.py`)
- Multiplex tracking → Mask decoder: uses decoder (evidence: `sam3/model/video_tracking_multiplex.py`)
- Multiplex tracking → SAM transformer: uses transformer (evidence: `sam3/model/video_tracking_multiplex.py`)
- Geometry encoding → Box operations: uses box operations (evidence: `sam3/model/geometry_encoders.py`)
- Geometry encoding → Model utilities: uses utilities (evidence: `sam3/model/geometry_encoders.py`)
- User → Agent inference: requests segmentation
- Agent inference → Agent orchestration: invokes agent (evidence: `sam3/agent/inference.py`)
- Agent orchestration → LLM client: uses language client
- Agent orchestration → SAM client: uses segmentation client
- Trainer → Training datasets: uses datasets (evidence: `sam3/train/trainer.py`)
- Trainer → Training losses: computes loss
- Evaluator → Evaluation suite: runs evaluation
- Evaluator → SA-Co video evaluation: evaluates video
- SA-Co video evaluation → Evaluation suite: uses metric tooling (evidence: `sam3/eval/saco_veval_evaluators.py`)
