# text-to-audio/audiolcm architecture

> Architecture diagram of the GitHub repository text-to-audio/audiolcm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/text-to-audio/audiolcm
- Repository: https://github.com/text-to-audio/audiolcm

## Overview

##### Project Type and Purpose
- This is a research-grade, full-stack deep-learning framework for text-to-audio generation based on Latent Consistency Models (AudioLCM).
- It provides both training pipelines (VAE + Latent Diffusion) and inference pipelines (single/batch), plus an integrated neural vocoder (BigVGAN) and evaluation scripts.

##### File Structure (Top-Level Components)
- configs/
– YAML configs for VAE, diffusion, teacher models, data paths, training hyperparams.
- ldm/
– data/ (TSV dataset readers, preprocessing scripts)
– models/ (autoencoder variants, diffusion samplers, schedulers)
– modules/ (attention, encoders: CLAP & OpenCLAP, discriminators, losses)
– util.py, lr_scheduler.py
- main.py
– Entrypoint for training (Hydra style: python main.py --base configs/… )
- pythonscripts/ & scripts/
– Wrappers for inference: InferAPI.py, txt2audio_for_lcm.py, batch scripts
- vocoder/bigvgan/
– BigVGAN code for waveform synthesis from mel spectrograms
- wav_evaluation/
– CLAP, FAD score calculators for evaluating generated audio
- requirements.txt, README.md

##### README for Additional Insights
- Training stages:
(1) Preprocess audio → melspec → TSV files
(2) Train VAE (autoencoder1d)
(3) Train Latent Diffusion (AudioLCM)
- Inference stages:
– Text prompt → AudioLCMBatchInfer/InferAPI → Latent consistency model → Latent output → Vocoder → .wav
- External dependencies: CLAP, T5, BERT, pretrained BigVGAN, HuggingFace models

##### Main Components & Interactions
A. Data Ingestion & Preprocessing
– Raw audio + text captions (TSV files)
– Preprocess scripts (mel_spec.py, add_duration.py) → stores mel-spectrograms, updated TSV
B. Dataset & DataLoader Modules
– joinaudiodataset_*.py, tsvdataset.py → yield (mel, text embeddings)
C. Training Pipeline
1. VAE Stage
- Encoder (autoencoder1d) ↔ Decoder
- Learns latent audio representation
2. Latent Diffusion Stage
- Text Encoder (CLAPWrapper, OpenCLAP modules) → text embeddings
- Diffusion model (ddpm_audio, ddim, lcm_audio, etc.) → latent generation
- Scheduler (ddim_solver, scheduling_lcm)
- Loss modules (diffusion loss, perceptual losses)
D. Inference Pipeline
– InferAPI.py / scripts/txt2audio_for_lcm.py
– Text prompt → Text encoder → Latent Consistency Model → Latent
– VAE Decoder (first stage) → Mel
– BigVGAN Vocoder → Waveform
E. Vocoder Component
– BigVGAN Inference: inference.py, inference_e2e.py → mels to audio
F. Evaluation Component
– wav_evaluation/cal_clap_score.py, cal_fad_score.py → compute metrics using CLAPWrapper and FAD metrics
G. Configuration & Orchestration
– Hydra (main.py with --base configs/…)
– Logging (logs/, ckpts/)

##### Architectural Patterns & Design Principles
- Modular separation: data, model, modules (encoder/discriminator/loss), scripts, evaluation
- Pipeline stages: clear two-stage training (VAE then diffusion)
- Config-driven (Hydra) for reproducibility
- Reusability: inference API separate from training code
- Pluggable encoders: CLAP, open_clap, T5, BERT

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers
    subgraph "Data & Preprocessing"
        direction TB
        RawAudio["Raw Audio & Text Caption"]:::data
        MelSpec["mel_spec.py"]:::data
        AddDur["add_duration.py"]:::data
        NATMel["NAT_mel.py"]:::data
        ProcessedData["Processed TSV & Mel-Spectrograms"]:::data
    end

    subgraph "Dataset & DataLoader"
        direction TB
        TSVData["tsvdataset.py"]:::data
        JoinAudio["joinaudiodataset_struct.py"]:::data
        DatasetOut["(mel, text embedding)"]:::data
    end

    subgraph "Configuration & Orchestration"
        direction TB
        Configs["configs/"]:::config
        Main["main.py"]:::config
    end

    subgraph "Training Pipeline"
        direction TB
        subgraph "VAE Stage"
            direction TB
            AE1["autoencoder1d.py"]:::deep
            AE2["autoencoder.py"]:::deep
            AE3["autoencoder_multi.py"]:::deep
            VAEOut["Latent Representation"]:::deep
        end
        subgraph "Latent Diffusion Stage"
            direction TB
            TextEnc["Text Encoders"]:::deep
            DDPM["ddpm_audio.py"]:::deep
            LCM["lcm_audio.py"]:::deep
            DDIM["ddim.py"]:::deep
            Solver["ddim_solver.py"]:::deep
            Scheduler["scheduling_lcm.py"]:::deep
            DiffOut["Latent Audio"]:::deep
        end
    end

    subgraph "Inference & Deployment"
        direction TB
        InferAPI["InferAPI.py"]:::infer
        TXT2Audio["txt2audio_for_lcm.py"]:::infer
        TextEncServer["Text Encoder"]:::infer
        LCMInfer["Latent Consistency Model"]:::infer
        VAEDecoder["VAE Decoder"]:::infer
        MelOut["Mel-Spectrogram"]:::infer
        Vocoder["BigVGAN Vocoder"]:::infer
        Waveform["Generated .wav"]:::infer
    end

    subgraph "Evaluation"
        direction TB
        CalCLAP["cal_clap_score.py"]:::eval
        CalFAD["cal_fad_score.py"]:::eval
        EvalOut["Evaluation Metrics"]:::eval
    end

    %% Connections Data
    RawAudio -->|".wav & captions"| MelSpec
    MelSpec --> AddDur --> NATMel
    NATMel -->|".tsv & mel"| ProcessedData
    ProcessedData --> TSVData
    ProcessedData --> JoinAudio
    TSVData --> DatasetOut
    JoinAudio --> DatasetOut

    %% Config to training
    Configs --> Main
    Main --> AE1
    Main --> DDPM

    %% VAE Stage
    DatasetOut --> AE1
    DatasetOut --> AE2
    DatasetOut --> AE3
    AE3 --> VAEOut

    %% Latent Diffusion Stage
    VAEOut --> DDPM
    DDPM --> Solver
    Solver --> LCM
    LCM --> Scheduler
    TextEnc --> LCM
    Scheduler --> DiffOut

    %% Inference Pipeline
    TXT2Audio --> TextEncServer
    InferAPI --> TextEncServer
    TextEncServer --> LCMInfer
    LCMInfer --> VAEDecoder
    VAEDecoder --> MelOut
    MelOut --> Vocoder
    Vocoder --> Waveform

    %% Evaluation
    Waveform --> CalCLAP
    Waveform --> CalFAD
    CalCLAP --> EvalOut
    CalFAD --> EvalOut

    %% Click Events
    click MelSpec "https://github.com/text-to-audio/audiolcm/blob/master/ldm/data/preprocess/mel_spec.py"
    click AddDur "https://github.com/text-to-audio/audiolcm/blob/master/ldm/data/preprocess/add_duration.py"
    click NATMel "https://github.com/text-to-audio/audiolcm/blob/master/ldm/data/preprocess/NAT_mel.py"
    click TSVData "https://github.com/text-to-audio/audiolcm/blob/master/ldm/data/tsvdataset.py"
    click JoinAudio "https://github.com/text-to-audio/audiolcm/blob/master/ldm/data/joinaudiodataset_struct.py"
    click AE1 "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/autoencoder1d.py"
    click AE2 "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/autoencoder.py"
    click AE3 "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/autoencoder_multi.py"
    click DDPM "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/diffusion/ddpm_audio.py"
    click LCM "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/diffusion/lcm_audio.py"
    click DDIM "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/diffusion/ddim.py"
    click Scheduler "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/diffusion/scheduling_lcm.py"
    click Solver "https://github.com/text-to-audio/audiolcm/blob/master/ldm/models/diffusion/ddim_solver.py"
    click TextEnc "https://github.com/text-to-audio/audiolcm/blob/master/ldm/modules/encoders/CLAP/CLAPWrapper.py"
    click TextEnc "https://github.com/text-to-audio/audiolcm/blob/master/ldm/modules/encoders/open_clap/factory.py"
    click Configs "https://github.com/text-to-audio/audiolcm/tree/master/configs/"
    click Main "https://github.com/text-to-audio/audiolcm/blob/master/main.py"
    click InferAPI "https://github.com/text-to-audio/audiolcm/blob/master/pythonscripts/InferAPI.py"
    click TXT2Audio "https://github.com/text-to-audio/audiolcm/blob/master/pythonscripts/txt2audio_for_lcm.py"
    click Vocoder "https://github.com/text-to-audio/audiolcm/blob/master/vocoder/bigvgan/inference.py"
    click Vocoder "https://github.com/text-to-audio/audiolcm/blob/master/vocoder/bigvgan/inference_e2e.py"
    click CalCLAP "https://github.com/text-to-audio/audiolcm/blob/master/wav_evaluation/cal_clap_score.py"
    click CalFAD "https://github.com/text-to-audio/audiolcm/blob/master/wav_evaluation/cal_fad_score.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef deep fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef infer fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef eval fill:#e1bee7,stroke:#8e24aa,color:#4a148c
    classDef config fill:#f0f4c3,stroke:#afb42b,color:#827717
```
