How facebookresearch/audiocraft works
Purpose and Entry Points
AudioCraft is a Python/PyTorch research library for generative audio: text-to-music/sound, neural audio compression, diffusion decoding, watermarking, and conditioned music generation. Primary programmatic inference APIs are model wrappers under audiocraft/models/musicgen.py, audiocraft/models/audiogen.py, audiocraft/models/jasco.py, and audiocraft/models/magnet.py. Training begins at audiocraft/train.py; interactive examples are notebook and app demos in demos/.
Model Facade and Checkpoint Loading
Task-facing model classes encapsulate generation setup, conditioning, codec interaction, and decoding. Shared construction/loading boundaries are centralized in audiocraft/models/builders.py, audiocraft/models/loaders.py, and audiocraft/models/genmodel.py. Pretrained weights are fetched through Hugging Face-backed caching, configurable via AUDIOCRAFT_CACHE_DIR; some melody workflows also depend on Demucs through Torch Hub.
Discrete Audio Representation
EnCodec converts waveform audio into quantized latent codebooks and reconstructs audio, enabling language-model-based generation over audio tokens. Its model implementation and vector quantization layer are in audiocraft/models/encodec.py, audiocraft/quantization/vq.py, and audiocraft/quantization/core_vq.py. Multi-band diffusion provides an alternate EnCodec-compatible decoder in audiocraft/models/multibanddiffusion.py, with core diffusion/UNet components in audiocraft/models/unet.py.
Generative Token and Diffusion Models
Autoregressive MusicGen/AudioGen language models are built around audiocraft/models/lm.py and transformer modules in audiocraft/modules/transformer.py. MAGNeT supplies non-autoregressive masked-token generation via audiocraft/models/lm_magnet.py and audiocraft/models/magnet.py. Flow/diffusion-oriented generation is represented by audiocraft/models/flow_matching.py; model families share conditioning and codec boundaries rather than a single serving runtime.
Conditioning Pipeline
Text, melody/chroma, style, chords, and drum inputs become conditioning representations before generation. General and JASCO-specific conditioners are implemented in audiocraft/modules/conditioners.py and audiocraft/modules/jasco_conditioners.py; JASCO’s composed model lives in audiocraft/models/jasco.py. Declarative conditioner selections define task capabilities, e.g. config/conditioner/text2music.yaml, config/conditioner/style2music.yaml, and config/conditioner/jasco_chords_drums_melody.yaml.
Data, Training, and Evaluation Control Plane
Audio datasets load and normalize waveform/metadata inputs through audiocraft/data/audio_dataset.py, with specialized music, sound, and JASCO dataset adapters. audiocraft/train.py resolves configuration into task-specific solvers; shared orchestration is in audiocraft/solvers/base.py, while family solvers include audiocraft/solvers/musicgen.py and audiocraft/solvers/compression.py. Solvers own training/evaluation loops, losses, checkpoints, distributed execution, and model-specific optimization.
Configuration and Compute Runtime
Hierarchical YAML configuration selects datasets, model structure, solver behavior, and experiment variants. Base composition begins at config/config.yaml; reproducible experiment definitions are represented by audiocraft/grids/musicgen/musicgen_base_32khz.py and analogous grids for compression, MAGNeT, diffusion, and watermarking. Runtime requires Python 3.9 and PyTorch 2.1.0; FFmpeg is materially required for audio I/O. Distributed/FSDP support is in audiocraft/optim/fsdp.py, and checkpoint/state handling in audiocraft/utils/checkpoint.py.
Watermarking and Quality Boundaries
AudioSeal watermark generation/detection spans audiocraft/models/watermark.py, audiocraft/modules/watermark.py, and its dedicated solver audiocraft/solvers/watermark.py. Audio quality, reconstruction, and generation objectives are separated into losses and metrics, including audiocraft/losses/specloss.py and audiocraft/metrics/fad.py; adversarial codec training uses discriminators under audiocraft/adversarial/discriminators/.
Main entry points: base.py Solver orchestration · conditioners.py Conditioning pipeline · lm.py Token language models · encodec.py EnCodec
Components 19
Inference APIs
- Task model facades · MusicGen, AudioGen, JASCO, and MAGNeT generation wrappers.
audiocraft/models/musicgen.py - Model build & loading · Constructs models and loads cached pretrained checkpoints.
audiocraft/models/builders.py - Pretrained checkpoints · Hugging Face-backed weights cache, configurable by AUDIOCRAFT_CACHE_DIR.
Generation & Audio Core
- Conditioning pipeline · Encodes text, melody, style, chords, and drums.
audiocraft/modules/conditioners.py - Conditioner selections · Declares conditioning capabilities for each task.
config/conditioner/text2music.yaml - Token language models · MusicGen and AudioGen predict discrete audio tokens.
audiocraft/models/lm.py - MAGNeT token model · Non-autoregressive audio-token generation.
audiocraft/models/lm_magnet.py - Flow matching
audiocraft/models/flow_matching.py - EnCodec · Maps waveforms to quantized codebooks and reconstructs audio.
audiocraft/models/encodec.py - Vector quantization
audiocraft/quantization/vq.py - Multi-band diffusion decoder · Alternate EnCodec-compatible waveform decoder.
audiocraft/models/multibanddiffusion.py
Training Control Plane
- Audio datasets · Loads and normalizes waveform and metadata inputs.
audiocraft/data/audio_dataset.py - Training entry point
audiocraft/train.py - Config & experiment grids · Composes dataset, model, solver, and reproducible experiment variants.
config/config.yaml - Solver orchestration · Owns training/evaluation loops, optimization, and distributed execution.
audiocraft/solvers/base.py - FSDP & checkpoint runtime
audiocraft/optim/fsdp.py
Quality & Watermarking
- AudioSeal watermarking · Generates and detects audio watermarks.
audiocraft/models/watermark.py - Watermark solver
audiocraft/solvers/watermark.py - Losses & quality metrics · Reconstruction, generation, and adversarial quality measurement.
audiocraft/losses/specloss.py
Connections 23
- Pretrained checkpoints → Model build & loading: loads weights
- Model build & loading → Task model facades: constructs
- Conditioner selections → Conditioning pipeline: selects
- Task model facades → Conditioning pipeline: prepares inputs
- Conditioning pipeline → Token language models: conditions
- Conditioning pipeline → MAGNeT token model: conditions
- Conditioning pipeline → Flow matching: conditions
- Task model facades → Token language models: generates tokens
- Task model facades → MAGNeT token model: generates tokens
- Token language models → EnCodec: decodes codebooks
- MAGNeT token model → EnCodec: decodes codebooks
- EnCodec → Vector quantization: quantizes latents
- Token language models → Multi-band diffusion decoder: alternate decode
- Multi-band diffusion decoder → EnCodec: codec-compatible output
- Config & experiment grids → Training entry point: composes run
- Training entry point → Solver orchestration: resolves solver
- Audio datasets → Solver orchestration: training data
- Solver orchestration → Token language models: trains
- Solver orchestration → EnCodec: trains
- Solver orchestration → FSDP & checkpoint runtime: distributed state
- Losses & quality metrics → Solver orchestration: losses and evaluation
- Watermark solver → AudioSeal watermarking: trains
- Losses & quality metrics → Watermark solver: evaluates