# funaudiollm/cosyvoice architecture

> Architecture diagram of the GitHub repository funaudiollm/cosyvoice, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/funaudiollm/cosyvoice
- Repository: https://github.com/funaudiollm/cosyvoice

## Overview

##### Project Type and Purpose
- It integrates several deep learning modules (e.g., transformer for TTS, HiFiGAN as a neural vocoder), and supports various interfaces (CLI, REST with FastAPI, gRPC, and even a web demo).
- The README and examples highlight its multilingual, low-latency, and fine-grained control features.

##### File Structure
- The top-level directories include:
- “cosyvoice”: The main package containing model scripts, CLI commands, dataset processing, flow control, and specialized modules (e.g., “transformer”, “hifigan”, “llm”, “tokenizer”, “dataset”).
- “asset”: Contains audio samples used for prompts in various inference examples.
- “runtime/python”: Contains deployment server scripts for both FastAPI and gRPC, which serve as API endpoints.
- “examples”: Provides ready-to-run configurations and scripts demonstrating usage with different datasets (e.g., LibriTTS and MagicData).
- “docker”: For deployment containerization.
- Other directories include “third_party”, “tools”, and GitHub configuration files.
- The extensive modularization indicates a layered architecture, with clear separation between core algorithms (encoding, tokenizing, neural network modules) and deployment/inference wrappers.

##### README for Additional Insights
- It provides usage examples that call the CLI functions and delineate how the inference modules combine with pre-trained models.
- Deployment instructions (Docker commands, FastAPI, and gRPC setups) reveal that the project is designed for both local testing and service deployment.
- The reference to borrowed codebases (e.g., FunASR, Matcha-TTS) further indicates the project’s hybrid nature, reusing proven modules for TTS tasks.

a. Main Components:
- User Interfaces (Input/Output):
- Command Line Interface (CLI via cosyvoice/cli)
- API Servers (FastAPI & gRPC from runtime/python)
- Web Demo (webui.py)
- Core TTS Engine:
- Inference and Training Scripts (in “cosyvoice/bin” and “examples”)
- Modules:
- Tokenizer (cosyvoice/tokenizer)
- Dataset Processing (cosyvoice/dataset)
- Flow Control (cosyvoice/flow) – handles decoders, length regulators, etc.
- Transformer-based Text Processing (cosyvoice/transformer)
- Vocoder Module (cosyvoice/hifigan)
- Large Language Model Integration (cosyvoice/llm)
- Utility and Support Libraries (cosyvoice/utils)
- Data & Model Resources:
- Pre-trained Models (downloadable and organized in “pretrained_models” via provided scripts)
- External libraries and third-party code (e.g., Matcha-TTS in third_party)
- Deployment Infrastructure:
- Docker configuration (“docker/” and “runtime/python/Dockerfile”)
- API server orchestration for scalable deployment

b. Relationships and Interactions:
- From User Input (CLI/API/Web UI) to the TTS Engine.
- Inside the TTS engine, the text data flows into tokenization, then through the transformer and flow modules, and finally through the HiFiGAN vocoder for audio synthesis.
- Interactions with external resources (pre-trained models, third-party libraries).
- Deployment components (Docker containers) wrap around the core API servers (FastAPI/gRPC) to expose interfaces externally.
- How training scripts and inference scripts interact with the dataset modules.
- Additionally, show how model export tools (export_jit.py, export_onnx.py, etc.) fit into the pipeline for serving or optimization.

c. Architectural Patterns:
- Layered architecture: User Interface → Application Logic (TTS Engine) → Data/Model Resources.
- Modular design: Clearly separated components (e.g., tokenizer, transformer, vocoder) allowing independent updates.
- Deployment pattern: Containerized microservices for API endpoints, isolating the inference/back-end logic from front-end clients.
- Pipeline style processing for inference that streams data from text input to final audio output and supports streaming and chunked processing.

d. Relevant Technologies & Frameworks:
- Python (core language, with modules and CLI)
- Deep learning frameworks (implied by scripts using PyTorch and torchaudio)
- FastAPI and gRPC (for API servers and client-server communication)
- Docker (containerization for deployment)
- External packages: “pynini”, “ttsfrd” (optional for text normalization and additional preprocessing)

- “Input Interfaces” (CLI, FastAPI, gRPC, Web UI)
- “Preprocessing & Tokenizer” (cosyvoice/tokenizer)
- “Text Transformer & Flow Modules” (cosyvoice/transformer, cosyvoice/flow)
- “Neural Vocoder” (cosyvoice/hifigan)
- “Model Management & Utilities” (dataset, utils, train/inference scripts)
- “Deployment/Containers” (Docker, runtime servers)
- Optionally, include notes on how pre-trained models are loaded and switched (CosyVoice2-0.5B, CosyVoice-300M variations).

##### Detail and Clarity
- Every module or directory that affects the TTS process is represented.
- Don’t shy away from splitting the core system into as many named components as possible, even if it means showing finer details (e.g., separating tokenizer from text normalization, separating different inference modes).
- Both static configurations (e.g., configuration files in “examples” and “docker”) and runtime components (API endpoints, flow processing).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Interfaces
    subgraph "Input Interfaces"
        CLI["Command Line Interface (CLI)"]:::interface
        FastAPI["API Server - FastAPI"]:::interface
        GRPC["API Server - gRPC"]:::interface
        WebDemo["Web Demo (webui.py)"]:::interface
    end

    %% Core TTS Engine
    subgraph "Core TTS Engine"
        Inference["Inference & Training Scripts"]:::engine
        ExportJIT["Export JIT"]:::engine
        ExportONNX["Export ONNX"]:::engine
        ExportTRT["Export TRT"]:::engine
        Tokenizer["Tokenizer Module"]:::engine
        Dataset["Dataset Processing"]:::engine
        Flow["Flow Control"]:::engine
        Transformer["Transformer Text Processing"]:::engine
        Vocal["Neural Vocoder Module"]:::engine
        LLM["Large Language Model Integration"]:::engine
        Utilities["Utility and Support Libraries"]:::engine
    end

    %% Deployment & Containerization
    subgraph "Deployment & Containerization"
        DockerConfig["Docker Configuration"]:::deployment
        ServerScripts["Server Scripts for Deployment"]:::deployment
    end

    %% External Dependencies
    subgraph "External Dependencies"
        ThirdParty["Third-party Codebase"]:::external
    end

    %% Data & Model Resources
    subgraph "Data & Model Resources"
        AudioSamples["Audio Samples"]:::data
    end

    %% Connections from Input Interfaces to Core Pipeline
    CLI -->|"CLI_Command"| Tokenizer
    FastAPI -->|"HTTP/REST"| Tokenizer
    GRPC -->|"gRPC"| Tokenizer
    WebDemo -->|"WebUI_input"| Tokenizer

    %% Core Processing Pipeline
    Tokenizer -->|"tokenization"| Transformer
    Transformer -->|"processing"| Flow
    Flow -->|"synthesis"| Vocal
    Vocal -->|"audio_output"| SynthAudio["Synthesized Audio"]
    Transformer -->|"enhances_with"| LLM

    %% Inference interactions within Core Engine
    Inference -->|"uses_data"| Dataset
    Inference -->|"utilizes"| Utilities
    Inference -->|"exports_model"| ExportJIT
    Inference -->|"exports_model"| ExportONNX
    Inference -->|"exports_model"| ExportTRT

    %% External Dependencies integration
    Transformer -->|"borrowed_code"| ThirdParty
    Vocal -->|"vocoder_boost"| ThirdParty

    %% Data & Model Resources
    AudioSamples -->|"provides_prompt"| Inference

    %% Deployment Connections
    DockerConfig -->|"containerizes"| ServerScripts
    ServerScripts -->|"hosts"| FastAPI
    ServerScripts -->|"hosts"| GRPC

    %% Click Events
    click CLI "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/cli"
    click FastAPI "https://github.com/funaudiollm/cosyvoice/tree/main/runtime/python/fastapi"
    click GRPC "https://github.com/funaudiollm/cosyvoice/tree/main/runtime/python/grpc"
    click WebDemo "https://github.com/funaudiollm/cosyvoice/blob/main/webui.py"
    click Inference "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/bin"
    click ExportJIT "https://github.com/funaudiollm/cosyvoice/blob/main/cosyvoice/bin/export_jit.py"
    click ExportONNX "https://github.com/funaudiollm/cosyvoice/blob/main/cosyvoice/bin/export_onnx.py"
    click ExportTRT "https://github.com/funaudiollm/cosyvoice/blob/main/cosyvoice/bin/export_trt.sh"
    click Tokenizer "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/tokenizer"
    click Dataset "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/dataset"
    click Flow "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/flow"
    click Transformer "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/transformer"
    click Vocal "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/hifigan"
    click LLM "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/llm"
    click Utilities "https://github.com/funaudiollm/cosyvoice/tree/main/cosyvoice/utils"
    click DockerConfig "https://github.com/funaudiollm/cosyvoice/tree/main/docker"
    click ServerScripts "https://github.com/funaudiollm/cosyvoice/tree/main/runtime/python"
    click ThirdParty "https://github.com/funaudiollm/cosyvoice/tree/main/third_party/Matcha-TTS"
    click AudioSamples "https://github.com/funaudiollm/cosyvoice/tree/main/asset"

    %% Styles
    classDef interface fill:#c2f0c2,stroke:#333,stroke-width:2px;
    classDef engine fill:#c2dff0,stroke:#333,stroke-width:2px;
    classDef deployment fill:#f0e68c,stroke:#333,stroke-width:2px;
    classDef external fill:#f4cccc,stroke:#333,stroke-width:2px;
    classDef data fill:#d9ead3,stroke:#333,stroke-width:2px;
```
