How neuphonic/neutts-air works
Project Type and Purpose
• NeuTTS Air is a Python-based, on-device text-to-speech (TTS) library that provides super-realistic speech synthesis with instant voice cloning.
• It exposes a simple Python API (NeuTTSAir class in neuttsair/neutts.py) and ships example scripts and notebooks for demonstration.
File Structure Analysis
• Top-Level
• neuttsair/ ← Core library module
• examples/ ← Sample client scripts & notebooks
• samples/ ← Pre-packaged reference audio/text pairs and model files
• requirements.txt ← Python dependencies (PyTorch, phonemizer, etc.)
• .pre-commit-config.yaml ← Code formatting & linting
• Patterns
• Library vs. consumer separation: neuttsair (the “backend”) vs. examples (the “frontend” clients)
• Optional decoders: ONNX (onnxruntime) or GGML (llama-cpp-python) tie into the same orchestration layer.
Key README Insights
• Two core sub-models:
a. Backbone LLM (Qwen 0.5B) for text encoding/generation
b. NeuCodec neural audio codec for waveform reconstruction
• External dependencies:
• espeak via phonemizer (for reference text → phoneme preprocessing)
• Model weights fetched from HuggingFace repos at runtime
• Deployment target: on-device (phones, embedded boards) using GGML/ONNX for performance.
Main Components & Relationships
1. Client Application
• Example scripts, notebooks, or your own service calling the NeuTTSAir API
• Supplies input text + reference audio/text
2. Orchestration Layer (NeuTTSAir class)
• Loads & configures:
• Text backbone (PyTorch or llama-cpp)
• Codec decoder (PyTorch, onnxruntime, or GGML)
• Phonemizer (espeak)
• Coordinates the end-to-end pipeline:
a. encode_reference() → audio → codec encoder → reference codes
b. infer() → text → backbone → codec decoder → waveform
3. Submodels
a. Backbone LLM
• Generates acoustic token representations conditioned on input text + reference codes
b. NeuCodec
• Encodes reference audio → discrete codes
• Decodes generated codes → raw audio waveform
4. External Services & Resources
• HuggingFace model repos (backbone_repo, codec_repo)
• Local dependencies (espeak library, phonemizer)
• Optional GPU/CUDA via llama-cpp-python or onnxruntime
Architectural Patterns & Design Principles
• Pipeline orchestration: clear stage separation (text → tokens → waveform)
• Pluggable backends: interchangeable decoder engines (PyTorch vs. ONNX vs. GGML)
• Single-responsibility modules: phonemizer, backbone, codec, CLI/examples