# ysharma3501/luxtts architecture

> Architecture diagram of the GitHub repository ysharma3501/luxtts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ysharma3501/luxtts
- Repository: https://github.com/ysharma3501/luxtts

## Overview

##### Project type and purpose
- This is a Python-based library and command-line toolkit for text-to-speech (TTS) model training, inference, and export.
- It implements a distilled version of ZipVoice (LuxTTS) for high-quality, low-latency voice cloning and generation at 48 kHz.

##### File structure
Top-level:
- zipvoice/ — core package
– bin/ — CLI scripts for dataset prep, training, inference, model export
– luxvoice.py — high-level API entrypoint (LuxTTS class)
– modeling_utils.py, onnx_modeling.py — helper functions for PyTorch/ONNX workflows
– models/ — neural architectures and modules: zipformer, two-stream, distill, dialog variants
– tokenizer/ — text/audio normalization and tokenization
– utils/ — checkpointing, feature extraction, diagnostics, LR scheduler, optimizers, TensorRT support
- pyproject.toml, requirements.txt — dependencies and build configuration
- README.md — usage instructions and feature overview

##### Key architectural insights from README
- Core features: voice cloning, high-quality 48 kHz output, fast inference (150× real time)
- Supports CPU/GPU/MPS, ONNX and TensorRT export
- Exposes simple Python API and multiple CLI entrypoints

##### Main components
a. Front-end (User interface layer)
- Python API (LuxTTS class in luxvoice.py)
- CLI scripts (zipvoice/bin/*.py)
b. Core library
- Tokenizer
- Feature extractor (compute_fbank, feature.py)
- Neural network models (zipformer modules, zipvoice.py, zipvoice_distill.py, zipvoice_dialog.py)
- Sampling engine (modeling_utils.py handles steps, t_shift, speed, return_smooth)
c. Output processing
- Vocoder integration (external Vocos or built-in 48 kHz vocoder)
- ONNX export (onnx_modeling.py)
- TensorRT export (tensorrt_export.py, utils/tensorrt.py)
d. Training pipeline
- Dataset preparation (prepare_dataset.py, prepare_tokens.py)
- Training scripts (train_zipvoice*.py)
- Checkpointing, scheduling, optimizer (utils/hooks.py, lr_scheduler.py, optim.py, checkpoint.py)
e. External services
- Hugging Face Model Hub (loading/saving pretrained weights)
- Optional Colab/Spaces deployment

##### Relationships and data flow
- User invokes CLI or Python API → Tokenizer normalizes text/audio → Feature extractor computes mel-filterbanks → Model forward pass → Sampling produces latent spectrogram → Vocoder decodes to waveform → Output written to .wav or returned to user
- Training: raw audio/text dataset → Dataset prep → Batch loader → Model + loss function → Optimizer + LR scheduler → Checkpoints
- Export: trained PyTorch model → ONNX export → TensorRT engine build

##### Architectural patterns and design principles
- Modular separation: tokenizer, modeling, utils, export, CLI scripts
- Single entrypoint API (LuxTTS) wrapping low-level modules
- Extensibility: export hooks for ONNX/TensorRT, support for different vocoders
- Configuration via function arguments (device, threads, sampling params)

##### Essential elements
- Clear component labels (e.g., “prepare_dataset.py”, “train_zipvoice.py”, “luxvoice.py”)
- Annotations for key parameters (device, num_steps, t_shift, speed)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interface Layer
    subgraph "Interface Layer"
        direction TB
        API["LuxTTS API"]:::interface
        CLI_CFB["compute_fbank"]:::interface
        CLI_GAM["generate_averaged_model"]:::interface
        CLI_INF_PT["infer_zipvoice (PyTorch)"]:::interface
        CLI_INF_DLG["infer_zipvoice_dialog"]:::interface
        CLI_INF_ONNX["infer_zipvoice_onnx"]:::interface
        CLI_ONNX_EXP["onnx_export"]:::interface
        CLI_TRT_EXP["tensorrt_export"]:::interface
        CLI_PREP_DS["prepare_dataset"]:::interface
        CLI_PREP_TOK["prepare_tokens"]:::interface
        CLI_TRN_BASE["train_zipvoice"]:::interface
        CLI_TRN_DLG["train_zipvoice_dialog"]:::interface
        CLI_TRN_DLG_ST["train_zipvoice_dialog_stereo"]:::interface
        CLI_TRN_DST["train_zipvoice_distill"]:::interface
    end

    %% Core Processing Layer
    subgraph "Core Processing Layer"
        direction TB
        subgraph Tokenizer
            direction TB
            NORMALIZER["normalizer"]:::util
            TOKENIZER_CORE["tokenizer core"]:::util
        end
        FEAT_EXT["Feature Extraction"]:::util
        subgraph Models
            direction TB
            ZIPFORMER["zipformer modules"]:::util
            TWO_STREAM["two-stream module"]:::util
            SCALING["scaling module"]:::util
            SOLVER["solver module"]:::util
            BASE_MODEL["base TTS model"]:::util
            DLG_MODEL["dialog variant"]:::util
            DST_MODEL["distillation variant"]:::util
        end
        SAMPLING["Sampling Engine & Inference Utils"]:::inference
        VOCODER["Vocoder"]:::inference
    end

    %% Export Layer
    subgraph "Export Layer"
        direction TB
        ONNX_MODEL["ONNX Workflows"]:::export
        TRT_UTIL["TensorRT Workflows"]:::export
    end

    %% Training Layer
    subgraph "Training Layer"
        direction TB
        DATA_PREP["Dataset Preparation"]:::training
        TOKEN_PREP["Token Preparation"]:::training
        subgraph Training_Scripts
            direction TB
            TRN_BASE["train_zipvoice"]:::training
            TRN_DLG["train_zipvoice_dialog"]:::training
            TRN_DLG_ST["train_zipvoice_dialog_stereo"]:::training
            TRN_DST["train_zipvoice_distill"]:::training
        end
        subgraph Training_Utils
            direction TB
            CHECKPOINT["Checkpointing"]:::util
            SCHEDULER["LR Scheduler"]:::util
            OPTIM["Optimizers"]:::util
            HOOKS["Training Hooks"]:::util
        end
    end

    %% External Integrations
    subgraph "External Integrations"
        direction TB
        HF["Hugging Face Hub"]:::external
        COLAB["Colab/Spaces"]:::external
    end

    %% Data and Control Flows
    API -->|"text/audio"| NORMALIZER
    CLI_CFB -->|"audio"| FEAT_EXT
    CLI_INF_PT -->|"text/audio"| NORMALIZER
    CLI_INF_ONNX -->|"text/audio"| NORMALIZER

    NORMALIZER -->|"tokens"| TOKENIZER_CORE
    TOKENIZER_CORE -->|"token ids"| FEAT_EXT
    FEAT_EXT -->|"mel-filterbanks"| BASE_MODEL
    BASE_MODEL -->|"latent spectrogram"| SAMPLING
    ZIPFORMER --> BASE_MODEL
    TWO_STREAM --> BASE_MODEL
    SCALING --> BASE_MODEL
    SOLVER --> BASE_MODEL
    DLG_MODEL --> BASE_MODEL
    DST_MODEL --> BASE_MODEL

    SAMPLING -->|"spectrogram"| VOCODER
    VOCODER -->|"waveform (.wav)"| API

    %% Export flows
    API --> CLI_ONNX_EXP
    CLI_ONNX_EXP --> ONNX_MODEL
    ONNX_MODEL -->|"model.onnx"| HF
    API --> CLI_TRT_EXP
    CLI_TRT_EXP --> TRT_UTIL
    TRT_UTIL -->|"engine.plan"| HF

    %% Training flows
    CLI_PREP_DS --> DATA_PREP
    CLI_PREP_TOK --> TOKEN_PREP
    DATA_PREP --> Training_Scripts
    TOKEN_PREP --> Training_Scripts
    Training_Scripts --> BASE_MODEL
    Training_Scripts --> DLG_MODEL
    Training_Scripts --> DST_MODEL
    Training_Scripts --> CHECKPOINT
    CHECKPOINT --> HF
    Training_Scripts --> SCHEDULER
    Training_Scripts --> OPTIM
    Training_Scripts --> HOOKS

    %% External
    API --> HF
    CLI_PREP_DS --> COLAB

    %% Click Events
    click API "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/luxvoice.py"
    click CLI_CFB "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/compute_fbank.py"
    click CLI_GAM "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/generate_averaged_model.py"
    click CLI_INF_PT "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/infer_zipvoice.py"
    click CLI_INF_DLG "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/infer_zipvoice_dialog.py"
    click CLI_INF_ONNX "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/infer_zipvoice_onnx.py"
    click CLI_ONNX_EXP "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/onnx_export.py"
    click CLI_TRT_EXP "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/tensorrt_export.py"
    click CLI_PREP_DS "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/prepare_dataset.py"
    click CLI_PREP_TOK "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/prepare_tokens.py"
    click CLI_TRN_BASE "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/train_zipvoice.py"
    click CLI_TRN_DLG "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/train_zipvoice_dialog.py"
    click CLI_TRN_DLG_ST "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/train_zipvoice_dialog_stereo.py"
    click CLI_TRN_DST "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/bin/train_zipvoice_distill.py"
    click NORMALIZER "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/tokenizer/normalizer.py"
    click TOKENIZER_CORE "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/tokenizer/tokenizer.py"
    click FEAT_EXT "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/feature.py"
    click SAMPLING "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/modeling_utils.py"
    click ONNX_MODEL "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/onnx_modeling.py"
    click TRT_UTIL "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/tensorrt.py"
    click CHECKPOINT "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/checkpoint.py"
    click SCHEDULER "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/lr_scheduler.py"
    click OPTIM "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/optim.py"
    click HOOKS "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/utils/hooks.py"
    click ZIPFORMER "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/modules/zipformer.py"
    click TWO_STREAM "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/modules/zipformer_two_stream.py"
    click SCALING "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/modules/scaling.py"
    click SOLVER "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/modules/solver.py"
    click BASE_MODEL "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/zipvoice.py"
    click DLG_MODEL "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/zipvoice_dialog.py"
    click DST_MODEL "https://github.com/ysharma3501/luxtts/blob/master/zipvoice/models/zipvoice_distill.py"

    %% Styles
    classDef interface fill:#D5DBDB,stroke:#000;
    classDef inference fill:#27AE60,stroke:#000;
    classDef training fill:#2980B9,stroke:#000;
    classDef export fill:#E67E22,stroke:#000;
    classDef util fill:#7F8C8D,stroke:#000;
    classDef external fill:#95A5A6,stroke:#000;
```
