How facebookresearch/sam-audio works
SAM-Audio is a prompt-driven audio source-separation model: users provide audio plus text, visual, or time-span cues and receive an isolated target and residual. The main path preprocesses prompts and audio, runs multimodal separation, and can generate and rerank candidate outputs. The repository also provides quality rankers and a separate evaluation workflow. Edges reflect the README and source-index references or shown calls; unsampled implementation details are not inferred.
Main entry points: model.py SAMAudio · main.py Evaluation Runner · ranker.py Ranker Ensemble · metrics Evaluation Metrics
Components 20
Prompt and Input
- Audio Processor
sam_audio/processor.py - Prepared Batch
sam_audio/processor.py
Separation Model
- SAMAudio
sam_audio/model/model.py - Audio Codec
sam_audio/model/codec.py - Text Encoder
sam_audio/model/text_encoder.py - Vision Encoder
sam_audio/model/vision_encoder.py - Diffusion Transformer
sam_audio/model/transformer.py - Modal Alignment
sam_audio/model/align.py - Sequence Patcher
sam_audio/model/patcher.py - Target and Residual
sam_audio/model/model.py
Candidate Ranking
- CLAP Ranker
sam_audio/ranking/clap.py - ImageBind Ranker
sam_audio/ranking/imagebind.py - Judge Ranker
sam_audio/ranking/judge.py - Judge Model
sam_audio/model/judge.py - Ranker Ensemble
sam_audio/ranking/ranker.py
Evaluation
- Evaluation Datasets
eval/dataset - Evaluation Runner
eval/main.py - Evaluation Metrics
eval/metrics
Other components
- User
- Checkpoint / Model Hub
Connections 26
- User → Audio Processor: provides prompts
README.md - Audio Processor → Prepared Batch: prepares inputs
sam_audio/processor.py - User → SAMAudio: requests separation
README.md - Prepared Batch → SAMAudio: supplies batch
README.md - SAMAudio → Audio Codec: encodes audio
sam_audio/model/model.py - SAMAudio → Text Encoder: encodes text
sam_audio/model/model.py - SAMAudio → Vision Encoder: encodes video
sam_audio/model/model.py - SAMAudio → Modal Alignment: aligns modalities
sam_audio/model/model.py - SAMAudio → Diffusion Transformer: predicts separation
sam_audio/model/model.py - Diffusion Transformer → Sequence Patcher: uses patches
sam_audio/model/transformer.py - SAMAudio → Target and Residual: returns audio
README.md - SAMAudio → Ranker Ensemble: reranks candidates
README.md - Ranker Ensemble → CLAP Ranker: scores text match
sam_audio/ranking/__init__.py - Ranker Ensemble → ImageBind Ranker: scores visual match
sam_audio/ranking/__init__.py - Ranker Ensemble → Judge Ranker: scores quality
sam_audio/ranking/__init__.py - Judge Ranker → Judge Model: evaluates candidates
sam_audio/ranking/judge.py - User → Evaluation Runner: runs evaluation
eval/main.py - Evaluation Runner → Evaluation Datasets: loads samples
eval/main.py - Evaluation Runner → Audio Processor: prepares batches
README.md - Evaluation Runner → SAMAudio: evaluates separation
eval/main.py - Evaluation Runner → Evaluation Metrics: computes metrics
eval/main.py - Evaluation Metrics → CLAP Ranker: reuses ranker
eval/metrics/clap.py - Evaluation Metrics → ImageBind Ranker: reuses transforms
eval/metrics/imagebind.py - Evaluation Metrics → Judge Model: scores quality
eval/metrics/judge.py - Audio Processor → Checkpoint / Model Hub: loads checkpoints
README.md - SAMAudio → Checkpoint / Model Hub: loads checkpoints
README.md