How nguyenthienhy/f5-tts-vietnamese works
Project type and purpose
• This is a Python-based machine-learning pipeline for fine-tuning and serving a Vietnamese Text-to-Speech (TTS) model based on the F5-TTS architecture.
• The README calls out two main use cases: (a) fine-tuning the model on custom audio/transcription data (via fine_tuning.sh), and (b) synthesizing speech from text or reference audio (via infer.sh or the CLI/Gradio interfaces).
File structure
• Top-level:
• Dockerfile, shell scripts (fine_tuning.sh, infer.sh), Python entry scripts (check_vocab_pretrained.py, convert_sr.py, etc.), pyproject.toml for packaging.
• data/: raw audio and vocabulary files.
• src/f5_tts/: the core Python package, subdivided into:
– configs/: YAML files defining model hyperparameters (E2TTS vs. F5TTS variants).
– model/: modules implementing backbones (DIT, MMDIT, UNETT), datasets, the trainer, and utility functions.
– train/: dataset preparation scripts and CLI/Gradio interfaces for fine-tuning.
– eval/: evaluation scripts and utilities for objective metrics.
– infer/: inference logic (CLI, Gradio, speech editing) plus examples.
– api.py, socket_server.py/socket_client.py: programmatic and network-based serving.
– scripts/: helper scripts (count params, epochs, etc.).
Extract insights from the README
• High-level pipeline: Data prep → embedding expansion → feature extraction → fine-tuning → inference.
• External dependencies: PyTorch (+ torchaudio), Whisper for ASR, sox/ffmpeg for audio I/O.
• Architecturally, there’s a single package exposing both training and serving entry points, plus shell wrappers for reproducibility.
Components and relationships
a. Main components
• Data Storage (raw audio files, transcripts)
• Preprocessing Module
• Vocabulary Manager (check & extend embeddings)
• Feature Extractor (spectrogram, pitch, etc.)
• Model Trainer (F5-TTS backbone + cfm + optimizer)
• Model Repository (saved checkpoints)
• Evaluation Module
• Inference Service
• CLI Interface
• Gradio Web UI
• Socket Server / Client
• External Services & Tools (Whisper ASR, sox, ffmpeg)
b. Interactions and data flow
1. Raw data → Preprocessing scripts (prepare_*) → formatted CSV / metadata
2. Metadata & vocab → Vocabulary Manager → updated embedding matrices
3. Feature Extractor reads audio based on metadata → tensor inputs
4. Model Trainer consumes features + configs → trains → writes checkpoints to Model Repository
5. Evaluation Module pulls checkpoints → runs test scripts → metrics
6. Inference Service loads checkpoint → uses Whisper ASR (if needed) + feature modules → synthesizes waveform → outputs audio file or streams via socket
7. CLI/Gradio wrap the Inference Service for user interaction
c. Architectural patterns & design principles
• Modular package structure for clear separation of concerns (configs, model, training, inference, evaluation).
• Pipeline orchestration via shell scripts for reproducibility.
• Plugin-style backbones: easy to swap DIT/MM-DIT/UNETT.
• Dual serving modes: local CLI/Gradio vs. networked socket server.
d. Key technologies & libraries
• PyTorch & torchaudio for modeling and audio ops
• Whisper ASR (openAI/whisper) for optional reference-text extraction
• sox, ffmpeg for audio processing
• YAML for config management
• Gradio for web-based inference GUI