# facebookresearch/audiocraft architecture

> Architecture diagram of the GitHub repository facebookresearch/audiocraft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/audiocraft
- Repository: https://github.com/facebookresearch/audiocraft

## Overview

#### Purpose and Entry Points
AudioCraft is a Python/PyTorch research library for generative audio: text-to-music/sound, neural audio compression, diffusion decoding, watermarking, and conditioned music generation. Primary programmatic inference APIs are model wrappers under `audiocraft/models/musicgen.py`, `audiocraft/models/audiogen.py`, `audiocraft/models/jasco.py`, and `audiocraft/models/magnet.py`. Training begins at `audiocraft/train.py`; interactive examples are notebook and app demos in `demos/`.

#### Model Facade and Checkpoint Loading
Task-facing model classes encapsulate generation setup, conditioning, codec interaction, and decoding. Shared construction/loading boundaries are centralized in `audiocraft/models/builders.py`, `audiocraft/models/loaders.py`, and `audiocraft/models/genmodel.py`. Pretrained weights are fetched through Hugging Face-backed caching, configurable via `AUDIOCRAFT_CACHE_DIR`; some melody workflows also depend on Demucs through Torch Hub.

#### Discrete Audio Representation
EnCodec converts waveform audio into quantized latent codebooks and reconstructs audio, enabling language-model-based generation over audio tokens. Its model implementation and vector quantization layer are in `audiocraft/models/encodec.py`, `audiocraft/quantization/vq.py`, and `audiocraft/quantization/core_vq.py`. Multi-band diffusion provides an alternate EnCodec-compatible decoder in `audiocraft/models/multibanddiffusion.py`, with core diffusion/UNet components in `audiocraft/models/unet.py`.

#### Generative Token and Diffusion Models
Autoregressive MusicGen/AudioGen language models are built around `audiocraft/models/lm.py` and transformer modules in `audiocraft/modules/transformer.py`. MAGNeT supplies non-autoregressive masked-token generation via `audiocraft/models/lm_magnet.py` and `audiocraft/models/magnet.py`. Flow/diffusion-oriented generation is represented by `audiocraft/models/flow_matching.py`; model families share conditioning and codec boundaries rather than a single serving runtime.

#### Conditioning Pipeline
Text, melody/chroma, style, chords, and drum inputs become conditioning representations before generation. General and JASCO-specific conditioners are implemented in `audiocraft/modules/conditioners.py` and `audiocraft/modules/jasco_conditioners.py`; JASCO’s composed model lives in `audiocraft/models/jasco.py`. Declarative conditioner selections define task capabilities, e.g. `config/conditioner/text2music.yaml`, `config/conditioner/style2music.yaml`, and `config/conditioner/jasco_chords_drums_melody.yaml`.

#### Data, Training, and Evaluation Control Plane
Audio datasets load and normalize waveform/metadata inputs through `audiocraft/data/audio_dataset.py`, with specialized music, sound, and JASCO dataset adapters. `audiocraft/train.py` resolves configuration into task-specific solvers; shared orchestration is in `audiocraft/solvers/base.py`, while family solvers include `audiocraft/solvers/musicgen.py` and `audiocraft/solvers/compression.py`. Solvers own training/evaluation loops, losses, checkpoints, distributed execution, and model-specific optimization.

#### Configuration and Compute Runtime
Hierarchical YAML configuration selects datasets, model structure, solver behavior, and experiment variants. Base composition begins at `config/config.yaml`; reproducible experiment definitions are represented by `audiocraft/grids/musicgen/musicgen_base_32khz.py` and analogous grids for compression, MAGNeT, diffusion, and watermarking. Runtime requires Python 3.9 and PyTorch 2.1.0; FFmpeg is materially required for audio I/O. Distributed/FSDP support is in `audiocraft/optim/fsdp.py`, and checkpoint/state handling in `audiocraft/utils/checkpoint.py`.

#### Watermarking and Quality Boundaries
AudioSeal watermark generation/detection spans `audiocraft/models/watermark.py`, `audiocraft/modules/watermark.py`, and its dedicated solver `audiocraft/solvers/watermark.py`. Audio quality, reconstruction, and generation objectives are separated into losses and metrics, including `audiocraft/losses/specloss.py` and `audiocraft/metrics/fad.py`; adversarial codec training uses discriminators under `audiocraft/adversarial/discriminators/`.

Main entry points: [`audiocraft/solvers/base.py`](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/solvers/base.py) (Solver orchestration), [`audiocraft/modules/conditioners.py`](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/modules/conditioners.py) (Conditioning pipeline), [`audiocraft/models/lm.py`](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/lm.py) (Token language models), [`audiocraft/models/encodec.py`](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/encodec.py) (EnCodec)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference APIs"]
  node_task_facades["Task model facades<br/>Python APIs<br/>[musicgen.py]"]
  node_model_loading["Model build &amp; loading<br/>model factory<br/>[builders.py]"]
  node_checkpoint_cache[("Pretrained checkpoints<br/>HF cache")]
end

subgraph group_generation["Generation &amp; Audio Core"]
  node_conditioners["Conditioning pipeline<br/>conditioning modules<br/>[conditioners.py]"]
  node_conditioner_configs["Conditioner selections<br/>YAML configs<br/>[text2music.yaml]"]
  node_token_lm["Token language models<br/>autoregressive transformer<br/>[lm.py]"]
  node_magnet_lm["MAGNeT token model<br/>masked-token model<br/>[lm_magnet.py]"]
  node_flow_model["Flow matching<br/>flow/diffusion model<br/>[flow_matching.py]"]
  node_encodec["EnCodec<br/>neural audio codec<br/>[encodec.py]"]
  node_vector_quantization["Vector quantization<br/>codebook layer<br/>[vq.py]"]
  node_diffusion_decoder["Multi-band diffusion decoder<br/>audio decoder"]
end

subgraph group_training["Training Control Plane"]
  node_datasets[("Audio datasets<br/>dataset pipeline<br/>[audio_dataset.py]")]
  node_train_entry["Training entry point<br/>Python entry point<br/>[train.py]"]
  node_experiment_config["Config &amp; experiment grids<br/>YAML/Python configs<br/>[config.yaml]"]
  node_solvers["Solver orchestration<br/>training framework<br/>[base.py]"]
  node_distributed_runtime["FSDP &amp; checkpoint runtime<br/>compute state<br/>[fsdp.py]"]
end

subgraph group_quality["Quality &amp; Watermarking"]
  node_watermarking["AudioSeal watermarking<br/>watermark model<br/>[watermark.py]"]
  node_watermark_solver["Watermark solver<br/>training solver<br/>[watermark.py]"]
  node_quality_objectives["Losses &amp; quality metrics<br/>evaluation objectives<br/>[specloss.py]"]
end

node_checkpoint_cache -->|"loads weights"| node_model_loading
node_model_loading -->|"constructs"| node_task_facades
node_conditioner_configs -->|"selects"| node_conditioners
node_task_facades -->|"prepares inputs"| node_conditioners
node_conditioners -->|"conditions"| node_token_lm
node_conditioners -->|"conditions"| node_magnet_lm
node_conditioners -->|"conditions"| node_flow_model
node_task_facades -->|"generates tokens"| node_token_lm
node_task_facades -->|"generates tokens"| node_magnet_lm
node_token_lm -->|"decodes codebooks"| node_encodec
node_magnet_lm -->|"decodes codebooks"| node_encodec
node_encodec -->|"quantizes latents"| node_vector_quantization
node_token_lm -.->|"alternate decode"| node_diffusion_decoder
node_diffusion_decoder -.->|"codec-compatible output"| node_encodec
node_experiment_config -->|"composes run"| node_train_entry
node_train_entry -->|"resolves solver"| node_solvers
node_datasets -->|"training data"| node_solvers
node_solvers -->|"trains"| node_token_lm
node_solvers -->|"trains"| node_encodec
node_solvers -->|"distributed state"| node_distributed_runtime
node_quality_objectives -->|"losses and evaluation"| node_solvers
node_watermark_solver -->|"trains"| node_watermarking
node_quality_objectives -->|"evaluates"| node_watermark_solver

click node_task_facades "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/musicgen.py"
click node_model_loading "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/builders.py"
click node_conditioners "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/modules/conditioners.py"
click node_conditioner_configs "https://github.com/facebookresearch/audiocraft/blob/main/config/conditioner/text2music.yaml"
click node_token_lm "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/lm.py"
click node_magnet_lm "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/lm_magnet.py"
click node_flow_model "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/flow_matching.py"
click node_encodec "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/encodec.py"
click node_vector_quantization "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/quantization/vq.py"
click node_diffusion_decoder "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/multibanddiffusion.py"
click node_datasets "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/data/audio_dataset.py"
click node_train_entry "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/train.py"
click node_experiment_config "https://github.com/facebookresearch/audiocraft/blob/main/config/config.yaml"
click node_solvers "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/solvers/base.py"
click node_distributed_runtime "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/optim/fsdp.py"
click node_watermarking "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/watermark.py"
click node_watermark_solver "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/solvers/watermark.py"
click node_quality_objectives "https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/losses/specloss.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task_facades,node_model_loading,node_checkpoint_cache toneBlue
class node_conditioners,node_conditioner_configs,node_token_lm,node_magnet_lm,node_flow_model,node_encodec,node_vector_quantization,node_diffusion_decoder toneAmber
class node_datasets,node_train_entry,node_experiment_config,node_solvers,node_distributed_runtime toneMint
class node_watermarking,node_watermark_solver,node_quality_objectives toneRose
```

## Components

### Inference APIs

- [Task model facades](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/musicgen.py): Python APIs · MusicGen, AudioGen, JASCO, and MAGNeT generation wrappers. · `audiocraft/models/musicgen.py`
- [Model build & loading](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/builders.py): model factory · Constructs models and loads cached pretrained checkpoints. · `audiocraft/models/builders.py`
- **Pretrained checkpoints**: HF cache · Hugging Face-backed weights cache, configurable by AUDIOCRAFT_CACHE_DIR.

### Generation & Audio Core

- [Conditioning pipeline](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/modules/conditioners.py): conditioning modules · Encodes text, melody, style, chords, and drums. · `audiocraft/modules/conditioners.py`
- [Conditioner selections](https://github.com/facebookresearch/audiocraft/blob/main/config/conditioner/text2music.yaml): YAML configs · Declares conditioning capabilities for each task. · `config/conditioner/text2music.yaml`
- [Token language models](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/lm.py): autoregressive transformer · MusicGen and AudioGen predict discrete audio tokens. · `audiocraft/models/lm.py`
- [MAGNeT token model](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/lm_magnet.py): masked-token model · Non-autoregressive audio-token generation. · `audiocraft/models/lm_magnet.py`
- [Flow matching](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/flow_matching.py): flow/diffusion model · `audiocraft/models/flow_matching.py`
- [EnCodec](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/encodec.py): neural audio codec · Maps waveforms to quantized codebooks and reconstructs audio. · `audiocraft/models/encodec.py`
- [Vector quantization](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/quantization/vq.py): codebook layer · `audiocraft/quantization/vq.py`
- [Multi-band diffusion decoder](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/multibanddiffusion.py): audio decoder · Alternate EnCodec-compatible waveform decoder. · `audiocraft/models/multibanddiffusion.py`

### Training Control Plane

- [Audio datasets](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/data/audio_dataset.py): dataset pipeline · Loads and normalizes waveform and metadata inputs. · `audiocraft/data/audio_dataset.py`
- [Training entry point](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/train.py): Python entry point · `audiocraft/train.py`
- [Config & experiment grids](https://github.com/facebookresearch/audiocraft/blob/main/config/config.yaml): YAML/Python configs · Composes dataset, model, solver, and reproducible experiment variants. · `config/config.yaml`
- [Solver orchestration](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/solvers/base.py): training framework · Owns training/evaluation loops, optimization, and distributed execution. · `audiocraft/solvers/base.py`
- [FSDP & checkpoint runtime](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/optim/fsdp.py): compute state · `audiocraft/optim/fsdp.py`

### Quality & Watermarking

- [AudioSeal watermarking](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/models/watermark.py): watermark model · Generates and detects audio watermarks. · `audiocraft/models/watermark.py`
- [Watermark solver](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/solvers/watermark.py): training solver · `audiocraft/solvers/watermark.py`
- [Losses & quality metrics](https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/losses/specloss.py): evaluation objectives · Reconstruction, generation, and adversarial quality measurement. · `audiocraft/losses/specloss.py`

## Connections

- Pretrained checkpoints → Model build & loading: loads weights
- Model build & loading → Task model facades: constructs
- Conditioner selections → Conditioning pipeline: selects
- Task model facades → Conditioning pipeline: prepares inputs
- Conditioning pipeline → Token language models: conditions
- Conditioning pipeline → MAGNeT token model: conditions
- Conditioning pipeline → Flow matching: conditions
- Task model facades → Token language models: generates tokens
- Task model facades → MAGNeT token model: generates tokens
- Token language models → EnCodec: decodes codebooks
- MAGNeT token model → EnCodec: decodes codebooks
- EnCodec → Vector quantization: quantizes latents
- Token language models → Multi-band diffusion decoder: alternate decode
- Multi-band diffusion decoder → EnCodec: codec-compatible output
- Config & experiment grids → Training entry point: composes run
- Training entry point → Solver orchestration: resolves solver
- Audio datasets → Solver orchestration: training data
- Solver orchestration → Token language models: trains
- Solver orchestration → EnCodec: trains
- Solver orchestration → FSDP & checkpoint runtime: distributed state
- Losses & quality metrics → Solver orchestration: losses and evaluation
- Watermark solver → AudioSeal watermarking: trains
- Losses & quality metrics → Watermark solver: evaluates
