# neuphonic/neutts-air architecture

> Architecture diagram of the GitHub repository neuphonic/neutts-air, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neuphonic/neutts-air
- Repository: https://github.com/neuphonic/neutts-air

## Overview

##### Project Type and Purpose
- NeuTTS Air is a Python-based, on-device text-to-speech (TTS) library that provides super-realistic speech synthesis with instant voice cloning.
- It exposes a simple Python API (`NeuTTSAir` class in neuttsair/neutts.py) and ships example scripts and notebooks for demonstration.

##### File Structure Analysis
- Top-Level
- neuttsair/              ← Core library module
- examples/               ← Sample client scripts & notebooks
- samples/                ← Pre-packaged reference audio/text pairs and model files
- requirements.txt        ← Python dependencies (PyTorch, phonemizer, etc.)
- .pre-commit-config.yaml ← Code formatting & linting
- Patterns
- Library vs. consumer separation: `neuttsair` (the “backend”) vs. `examples` (the “frontend” clients)
- Optional decoders: ONNX (`onnxruntime`) or GGML (`llama-cpp-python`) tie into the same orchestration layer.

##### Key README Insights
- Two core sub-models:
a. Backbone LLM (Qwen 0.5B) for text encoding/generation
b. NeuCodec neural audio codec for waveform reconstruction
- External dependencies:
- espeak via phonemizer (for reference text → phoneme preprocessing)
- Model weights fetched from HuggingFace repos at runtime
- Deployment target: on-device (phones, embedded boards) using GGML/ONNX for performance.

##### Main Components & Relationships
1. Client Application
- Example scripts, notebooks, or your own service calling the NeuTTSAir API
- Supplies input text + reference audio/text
2. Orchestration Layer (`NeuTTSAir` class)
- Loads & configures:
- Text backbone (PyTorch or llama-cpp)
- Codec decoder (PyTorch, onnxruntime, or GGML)
- Phonemizer (espeak)
- Coordinates the end-to-end pipeline:
a. encode_reference() → audio → codec encoder → reference codes
b. infer() → text → backbone → codec decoder → waveform
3. Submodels
a. Backbone LLM
- Generates acoustic token representations conditioned on input text + reference codes
b. NeuCodec
- Encodes reference audio → discrete codes
- Decodes generated codes → raw audio waveform
4. External Services & Resources
- HuggingFace model repos (backbone_repo, codec_repo)
- Local dependencies (espeak library, phonemizer)
- Optional GPU/CUDA via llama-cpp-python or onnxruntime

##### Architectural Patterns & Design Principles
- Pipeline orchestration: clear stage separation (text → tokens → waveform)
- Pluggable backends: interchangeable decoder engines (PyTorch vs. ONNX vs. GGML)
- Single-responsibility modules: phonemizer, backbone, codec, CLI/examples

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Applications
    subgraph "Client Applications" 
        direction TB
        BasicExample["basic_example.py"]:::external
        StreamingExample["basic_streaming_example.py"]:::external
        EncodeRefExample["encode_reference.py"]:::external
        InteractiveExample["interactive_example.ipynb"]:::external
        ONNXExample["onnx_example.py"]:::external
    end

    %% Orchestration Layer
    Orchestrator["NeuTTSAir (neuttsair/neutts.py)"]:::orchestrator

    %% Phonemizer
    Phonemizer["Phonemizer\n(espeak-ng)"]:::process

    %% Codec Encoder & Decoder
    subgraph "Audio Codec (NeuCodec)" 
        direction TB
        CodecEncoder["Encoder"]:::ML
        subgraph "Decoder Engines" 
            direction TB
            DecoderPyTorch["Decoder\n(PyTorch)"]:::ML
            DecoderONNX["Decoder\n(onnxruntime)"]:::ML
            DecoderGGML["Decoder\n(GGML)"]:::ML
        end
    end

    %% Text Backbone
    Backbone["Text Backbone\n(Qwen 0.5B)"]:::ML

    %% Output Waveform
    Output["Output Waveform"]:::process

    %% External Services & Repos
    subgraph "External Dependencies & Repos"
        direction TB
        HuggingFace["HuggingFace Model Repos"]:::external
        Requirements["requirements.txt"]:::external
        PreCommit[".pre-commit-config.yaml"]:::external
        SamplesDir["samples/"]:::external
    end

    %% Connections
    BasicExample -->|calls API| Orchestrator
    StreamingExample -->|calls API| Orchestrator
    EncodeRefExample -->|calls API| Orchestrator
    InteractiveExample -->|calls API| Orchestrator
    ONNXExample -->|calls API| Orchestrator

    %% Reference flow
    SamplesDir -->|reference audio/text| Phonemizer
    Phonemizer -->|phonemes| CodecEncoder
    CodecEncoder -->|codes| Orchestrator

    %% Text flow
    Orchestrator -->|text input| Backbone
    Backbone -->|acoustic tokens| Orchestrator

    %% Synthesis flow
    Orchestrator -->|generated codes| DecoderPyTorch
    Orchestrator -->|generated codes| DecoderONNX
    Orchestrator -->|generated codes| DecoderGGML
    DecoderPyTorch -->|waveform| Output
    DecoderONNX -.->|optional: onnxruntime| Output
    DecoderGGML -.->|optional: llama-cpp| Output

    %% External resources
    Orchestrator -->|fetch models| HuggingFace
    Orchestrator -->|uses deps| Requirements
    Orchestrator -->|uses deps| PreCommit

    %% Click Events
    click BasicExample "https://github.com/neuphonic/neutts-air/blob/main/examples/basic_example.py"
    click StreamingExample "https://github.com/neuphonic/neutts-air/blob/main/examples/basic_streaming_example.py"
    click EncodeRefExample "https://github.com/neuphonic/neutts-air/blob/main/examples/encode_reference.py"
    click InteractiveExample "https://github.com/neuphonic/neutts-air/blob/main/examples/interactive_example.ipynb"
    click ONNXExample "https://github.com/neuphonic/neutts-air/blob/main/examples/onnx_example.py"
    click Orchestrator "https://github.com/neuphonic/neutts-air/blob/main/neuttsair/neutts.py"
    click SamplesDir "https://github.com/neuphonic/neutts-air/tree/main/samples/"
    click Requirements "https://github.com/neuphonic/neutts-air/blob/main/requirements.txt"
    click PreCommit "https://github.com/neuphonic/neutts-air/blob/main/.pre-commit-config.yaml"

    %% Styles
    classDef orchestrator fill:#D6E4FF,stroke:#3B82F6,stroke-width:1px;
    classDef ML fill:#D1FAE5,stroke:#10B981,stroke-width:1px;
    classDef external fill:#FEF3C7,stroke:#F59E0B,stroke-width:1px;
    classDef process fill:#E5E7EB,stroke:#6B7280,stroke-width:1px;
```
