# facebookresearch/sam-audio architecture

> Architecture diagram of the GitHub repository facebookresearch/sam-audio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/sam-audio
- Repository: https://github.com/facebookresearch/sam-audio

## Overview

SAM-Audio is a prompt-driven audio source-separation model: users provide audio plus text, visual, or time-span cues and receive an isolated target and residual. The main path preprocesses prompts and audio, runs multimodal separation, and can generate and rerank candidate outputs. The repository also provides quality rankers and a separate evaluation workflow. Edges reflect the README and source-index references or shown calls; unsampled implementation details are not inferred.

Main entry points: [`sam_audio/model/model.py`](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/model.py) (SAMAudio), [`eval/main.py`](https://github.com/facebookresearch/sam-audio/blob/main/eval/main.py) (Evaluation Runner), [`sam_audio/ranking/ranker.py`](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/ranker.py) (Ranker Ensemble), [`eval/metrics`](https://github.com/facebookresearch/sam-audio/tree/main/eval/metrics) (Evaluation Metrics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Prompt and Input"]
  node_processor["Audio Processor<br/>[processor.py]"]
  node_batch["Prepared Batch<br/>[processor.py]"]
end

subgraph group_separation["Separation Model"]
  node_model["SAMAudio<br/>[model.py]"]
  node_codec["Audio Codec<br/>[codec.py]"]
  node_text["Text Encoder<br/>[text_encoder.py]"]
  node_vision["Vision Encoder<br/>[vision_encoder.py]"]
  node_transformer["Diffusion Transformer<br/>[transformer.py]"]
  node_alignment["Modal Alignment<br/>[align.py]"]
  node_patcher["Sequence Patcher<br/>[patcher.py]"]
  node_outputs["Target and Residual<br/>[model.py]"]
end

subgraph group_ranking["Candidate Ranking"]
  node_clap["CLAP Ranker<br/>[clap.py]"]
  node_imagebind["ImageBind Ranker<br/>[imagebind.py]"]
  node_judge_ranker["Judge Ranker<br/>[judge.py]"]
  node_judge_model["Judge Model<br/>[judge.py]"]
  node_ranker["Ranker Ensemble<br/>[ranker.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_dataset["Evaluation Datasets"]
  node_eval_runner["Evaluation Runner<br/>[main.py]"]
  node_eval_metrics["Evaluation Metrics"]
end

node_user(("User"))
node_external{{"Checkpoint / Model Hub"}}

node_user -->|"provides prompts"| node_processor
node_processor -->|"prepares inputs"| node_batch
node_user -->|"requests separation"| node_model
node_batch -->|"supplies batch"| node_model
node_model -->|"encodes audio"| node_codec
node_model -->|"encodes text"| node_text
node_model -->|"encodes video"| node_vision
node_model -->|"aligns modalities"| node_alignment
node_model -->|"predicts separation"| node_transformer
node_transformer -->|"uses patches"| node_patcher
node_model -->|"returns audio"| node_outputs
node_model -->|"reranks candidates"| node_ranker
node_ranker -->|"scores text match"| node_clap
node_ranker -->|"scores visual match"| node_imagebind
node_ranker -->|"scores quality"| node_judge_ranker
node_judge_ranker -->|"evaluates candidates"| node_judge_model
node_user -->|"runs evaluation"| node_eval_runner
node_eval_runner -->|"loads samples"| node_dataset
node_eval_runner -->|"prepares batches"| node_processor
node_eval_runner -->|"evaluates separation"| node_model
node_eval_runner -->|"computes metrics"| node_eval_metrics
node_eval_metrics -->|"reuses ranker"| node_clap
node_eval_metrics -->|"reuses transforms"| node_imagebind
node_eval_metrics -->|"scores quality"| node_judge_model
node_processor -->|"loads checkpoints"| node_external
node_model -->|"loads checkpoints"| node_external

click node_processor "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/processor.py"
click node_batch "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/processor.py"
click node_dataset "https://github.com/facebookresearch/sam-audio/tree/main/eval/dataset"
click node_model "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/model.py"
click node_codec "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/codec.py"
click node_text "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/text_encoder.py"
click node_vision "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/vision_encoder.py"
click node_transformer "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/transformer.py"
click node_alignment "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/align.py"
click node_patcher "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/patcher.py"
click node_clap "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/clap.py"
click node_imagebind "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/imagebind.py"
click node_judge_ranker "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/judge.py"
click node_judge_model "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/judge.py"
click node_ranker "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/ranker.py"
click node_eval_runner "https://github.com/facebookresearch/sam-audio/blob/main/eval/main.py"
click node_eval_metrics "https://github.com/facebookresearch/sam-audio/tree/main/eval/metrics"
click node_outputs "https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_processor,node_batch,node_user toneBlue
class node_model,node_codec,node_text,node_vision,node_transformer,node_alignment,node_patcher,node_outputs toneAmber
class node_clap,node_imagebind,node_judge_ranker,node_judge_model,node_ranker toneMint
class node_dataset,node_eval_runner,node_eval_metrics toneRose
class node_external toneIndigo
```

## Components

### Prompt and Input

- [Audio Processor](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/processor.py): `sam_audio/processor.py`
- [Prepared Batch](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/processor.py): `sam_audio/processor.py`

### Separation Model

- [SAMAudio](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/model.py): `sam_audio/model/model.py`
- [Audio Codec](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/codec.py): `sam_audio/model/codec.py`
- [Text Encoder](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/text_encoder.py): `sam_audio/model/text_encoder.py`
- [Vision Encoder](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/vision_encoder.py): `sam_audio/model/vision_encoder.py`
- [Diffusion Transformer](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/transformer.py): `sam_audio/model/transformer.py`
- [Modal Alignment](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/align.py): `sam_audio/model/align.py`
- [Sequence Patcher](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/patcher.py): `sam_audio/model/patcher.py`
- [Target and Residual](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/model.py): `sam_audio/model/model.py`

### Candidate Ranking

- [CLAP Ranker](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/clap.py): `sam_audio/ranking/clap.py`
- [ImageBind Ranker](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/imagebind.py): `sam_audio/ranking/imagebind.py`
- [Judge Ranker](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/judge.py): `sam_audio/ranking/judge.py`
- [Judge Model](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/model/judge.py): `sam_audio/model/judge.py`
- [Ranker Ensemble](https://github.com/facebookresearch/sam-audio/blob/main/sam_audio/ranking/ranker.py): `sam_audio/ranking/ranker.py`

### Evaluation

- [Evaluation Datasets](https://github.com/facebookresearch/sam-audio/tree/main/eval/dataset): `eval/dataset`
- [Evaluation Runner](https://github.com/facebookresearch/sam-audio/blob/main/eval/main.py): `eval/main.py`
- [Evaluation Metrics](https://github.com/facebookresearch/sam-audio/tree/main/eval/metrics): `eval/metrics`

### Other components

- **User**
- **Checkpoint / Model Hub**

## Connections

- User → Audio Processor: provides prompts (evidence: `README.md`)
- Audio Processor → Prepared Batch: prepares inputs (evidence: `sam_audio/processor.py`)
- User → SAMAudio: requests separation (evidence: `README.md`)
- Prepared Batch → SAMAudio: supplies batch (evidence: `README.md`)
- SAMAudio → Audio Codec: encodes audio (evidence: `sam_audio/model/model.py`)
- SAMAudio → Text Encoder: encodes text (evidence: `sam_audio/model/model.py`)
- SAMAudio → Vision Encoder: encodes video (evidence: `sam_audio/model/model.py`)
- SAMAudio → Modal Alignment: aligns modalities (evidence: `sam_audio/model/model.py`)
- SAMAudio → Diffusion Transformer: predicts separation (evidence: `sam_audio/model/model.py`)
- Diffusion Transformer → Sequence Patcher: uses patches (evidence: `sam_audio/model/transformer.py`)
- SAMAudio → Target and Residual: returns audio (evidence: `README.md`)
- SAMAudio → Ranker Ensemble: reranks candidates (evidence: `README.md`)
- Ranker Ensemble → CLAP Ranker: scores text match (evidence: `sam_audio/ranking/__init__.py`)
- Ranker Ensemble → ImageBind Ranker: scores visual match (evidence: `sam_audio/ranking/__init__.py`)
- Ranker Ensemble → Judge Ranker: scores quality (evidence: `sam_audio/ranking/__init__.py`)
- Judge Ranker → Judge Model: evaluates candidates (evidence: `sam_audio/ranking/judge.py`)
- User → Evaluation Runner: runs evaluation (evidence: `eval/main.py`)
- Evaluation Runner → Evaluation Datasets: loads samples (evidence: `eval/main.py`)
- Evaluation Runner → Audio Processor: prepares batches (evidence: `README.md`)
- Evaluation Runner → SAMAudio: evaluates separation (evidence: `eval/main.py`)
- Evaluation Runner → Evaluation Metrics: computes metrics (evidence: `eval/main.py`)
- Evaluation Metrics → CLAP Ranker: reuses ranker (evidence: `eval/metrics/clap.py`)
- Evaluation Metrics → ImageBind Ranker: reuses transforms (evidence: `eval/metrics/imagebind.py`)
- Evaluation Metrics → Judge Model: scores quality (evidence: `eval/metrics/judge.py`)
- Audio Processor → Checkpoint / Model Hub: loads checkpoints (evidence: `README.md`)
- SAMAudio → Checkpoint / Model Hub: loads checkpoints (evidence: `README.md`)