How dangvansam/viet-tts works
Main Components
• API Server & CLI: These are exposed through the “viettts/server.py” (for starting the API server) and “viettts/cli.py” (for the CLI commands). This component accepts text or audio inputs and returns synthesized speech.
• TTS Engine Pipeline: Divided into several parts:
– Text Preprocessing/Tokenization: Located in “viettts/tokenizer” (and its multilingual.tiktoken support) and possibly part of “viettts/utils/frontend_utils.py” for front-end text normalization.
– Transformer Module: Found under “viettts/transformer” with multiple submodules for encoder, decoder, attention, etc. This is the core that processes text into an intermediate representation.
– Flow Module: Located in “viettts/flow” contains the decoder, flow matching, and length regulator. This likely manages the timing/alignment aspects of text-to-speech conversion.
– Audio Generation (HifiGAN): Under “viettts/hifigan” with its generator and f0 predictor modules to convert processed representations into audio waveforms.
– Optional LLM Module: The “viettts/llm” directory, which could be used for additional language modeling or voice prompts.
• Utility Modules: “viettts/utils” providing helper functions for file I/O, common tasks, VAD (voice activity detection) and more.
• Pre-built Samples and Assets: The “samples” directory shows example audio files for built-in voices, and “assets” contains branding images, etc.
• Docker & Deployment files: Files like “Dockerfile”, “docker-compose.yaml”, and configurations under “.github/workflows” illustrate the CI/CD, containerization, and deployment processes.
Relationships and Flow
• User Interaction Layer:
– Users can interact via the CLI (command-line commands) or invoke REST API calls to the server.
• Request Flow:
– API/CLI receives a text input (and optionally an audio file for voice cloning).
– The input is passed to the text preprocessor/tokenizer to prepare it for synthesis.
– The transformer module then processes this tokenized input to generate an intermediate representation.
– The flow module adjusts timing and alignment before handing data off to the HifiGAN module.
– HifiGAN then synthesizes the final waveform.
– The synthesized audio is sent back as a response.
• Auxiliary interactions:
– Utility modules provide support across all stages.
– Docker configurations ensure that the API server and any required dependencies (e.g., NVIDIA drivers, CUDA) are properly containerized and deployed.
Architectural Patterns & Design Principles
• Modular Design: The project is organized into clearly separated components (tokenizer, transformer, flow, HifiGAN, CLI, server), enabling independent development and testing.
• Pipeline Architecture: The audio synthesis process follows a sequential pipeline, which can be visualized as a series of data transformations.
• Containerization & CI/CD: Docker and GitHub workflows indicate that the deployment and continuous integration/delivery are core parts of the development process.
• Extensibility: The design supports adding new voices and configurations easily, as seen in the “samples” directory and built-in voice commands.