How ysharma3501/luxtts works
Project type and purpose
• This is a Python-based library and command-line toolkit for text-to-speech (TTS) model training, inference, and export.
• It implements a distilled version of ZipVoice (LuxTTS) for high-quality, low-latency voice cloning and generation at 48 kHz.
File structure
Top-level:
• zipvoice/ — core package
– bin/ — CLI scripts for dataset prep, training, inference, model export
– luxvoice.py — high-level API entrypoint (LuxTTS class)
– modeling_utils.py, onnx_modeling.py — helper functions for PyTorch/ONNX workflows
– models/ — neural architectures and modules: zipformer, two-stream, distill, dialog variants
– tokenizer/ — text/audio normalization and tokenization
– utils/ — checkpointing, feature extraction, diagnostics, LR scheduler, optimizers, TensorRT support
• pyproject.toml, requirements.txt — dependencies and build configuration
• README.md — usage instructions and feature overview
Key architectural insights from README
• Core features: voice cloning, high-quality 48 kHz output, fast inference (150× real time)
• Supports CPU/GPU/MPS, ONNX and TensorRT export
• Exposes simple Python API and multiple CLI entrypoints
Main components
a. Front-end (User interface layer)
• Python API (LuxTTS class in luxvoice.py)
• CLI scripts (zipvoice/bin/*.py)
b. Core library
• Tokenizer
• Feature extractor (compute_fbank, feature.py)
• Neural network models (zipformer modules, zipvoice.py, zipvoice_distill.py, zipvoice_dialog.py)
• Sampling engine (modeling_utils.py handles steps, t_shift, speed, return_smooth)
c. Output processing
• Vocoder integration (external Vocos or built-in 48 kHz vocoder)
• ONNX export (onnx_modeling.py)
• TensorRT export (tensorrt_export.py, utils/tensorrt.py)
d. Training pipeline
• Dataset preparation (prepare_dataset.py, prepare_tokens.py)
• Training scripts (train_zipvoice*.py)
• Checkpointing, scheduling, optimizer (utils/hooks.py, lr_scheduler.py, optim.py, checkpoint.py)
e. External services
• Hugging Face Model Hub (loading/saving pretrained weights)
• Optional Colab/Spaces deployment
Relationships and data flow
• User invokes CLI or Python API → Tokenizer normalizes text/audio → Feature extractor computes mel-filterbanks → Model forward pass → Sampling produces latent spectrogram → Vocoder decodes to waveform → Output written to .wav or returned to user
• Training: raw audio/text dataset → Dataset prep → Batch loader → Model + loss function → Optimizer + LR scheduler → Checkpoints
• Export: trained PyTorch model → ONNX export → TensorRT engine build
Architectural patterns and design principles
• Modular separation: tokenizer, modeling, utils, export, CLI scripts
• Single entrypoint API (LuxTTS) wrapping low-level modules
• Extensibility: export hooks for ONNX/TensorRT, support for different vocoders
• Configuration via function arguments (device, threads, sampling params)
Essential elements
• Clear component labels (e.g., “prepare_dataset.py”, “train_zipvoice.py”, “luxvoice.py”)
• Annotations for key parameters (device, num_steps, t_shift, speed)