How text-to-audio/audiolcm works
Project Type and Purpose
• This is a research-grade, full-stack deep-learning framework for text-to-audio generation based on Latent Consistency Models (AudioLCM).
• It provides both training pipelines (VAE + Latent Diffusion) and inference pipelines (single/batch), plus an integrated neural vocoder (BigVGAN) and evaluation scripts.
File Structure (Top-Level Components)
• configs/
– YAML configs for VAE, diffusion, teacher models, data paths, training hyperparams.
• ldm/
– data/ (TSV dataset readers, preprocessing scripts)
– models/ (autoencoder variants, diffusion samplers, schedulers)
– modules/ (attention, encoders: CLAP & OpenCLAP, discriminators, losses)
– util.py, lr_scheduler.py
• main.py
– Entrypoint for training (Hydra style: python main.py --base configs/… )
• pythonscripts/ & scripts/
– Wrappers for inference: InferAPI.py, txt2audio_for_lcm.py, batch scripts
• vocoder/bigvgan/
– BigVGAN code for waveform synthesis from mel spectrograms
• wav_evaluation/
– CLAP, FAD score calculators for evaluating generated audio
• requirements.txt, README.md
README for Additional Insights
• Training stages:
(1) Preprocess audio → melspec → TSV files
(2) Train VAE (autoencoder1d)
(3) Train Latent Diffusion (AudioLCM)
• Inference stages:
– Text prompt → AudioLCMBatchInfer/InferAPI → Latent consistency model → Latent output → Vocoder → .wav
• External dependencies: CLAP, T5, BERT, pretrained BigVGAN, HuggingFace models
Main Components & Interactions
A. Data Ingestion & Preprocessing
– Raw audio + text captions (TSV files)
– Preprocess scripts (mel_spec.py, add_duration.py) → stores mel-spectrograms, updated TSV
B. Dataset & DataLoader Modules
– joinaudiodataset_*.py, tsvdataset.py → yield (mel, text embeddings)
C. Training Pipeline
1. VAE Stage
• Encoder (autoencoder1d) ↔ Decoder
• Learns latent audio representation
2. Latent Diffusion Stage
• Text Encoder (CLAPWrapper, OpenCLAP modules) → text embeddings
• Diffusion model (ddpm_audio, ddim, lcm_audio, etc.) → latent generation
• Scheduler (ddim_solver, scheduling_lcm)
• Loss modules (diffusion loss, perceptual losses)
D. Inference Pipeline
– InferAPI.py / scripts/txt2audio_for_lcm.py
– Text prompt → Text encoder → Latent Consistency Model → Latent
– VAE Decoder (first stage) → Mel
– BigVGAN Vocoder → Waveform
E. Vocoder Component
– BigVGAN Inference: inference.py, inference_e2e.py → mels to audio
F. Evaluation Component
– wav_evaluation/cal_clap_score.py, cal_fad_score.py → compute metrics using CLAPWrapper and FAD metrics
G. Configuration & Orchestration
– Hydra (main.py with --base configs/…)
– Logging (logs/, ckpts/)
Architectural Patterns & Design Principles
• Modular separation: data, model, modules (encoder/discriminator/loss), scripts, evaluation
• Pipeline stages: clear two-stage training (VAE then diffusion)
• Config-driven (Hydra) for reproducibility
• Reusability: inference API separate from training code
• Pluggable encoders: CLAP, open_clap, T5, BERT