# nguyenthienhy/f5-tts-vietnamese architecture

> Architecture diagram of the GitHub repository nguyenthienhy/f5-tts-vietnamese, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nguyenthienhy/f5-tts-vietnamese
- Repository: https://github.com/nguyenthienhy/f5-tts-vietnamese

## Overview

##### Project type and purpose
- This is a Python-based machine-learning pipeline for fine-tuning and serving a Vietnamese Text-to-Speech (TTS) model based on the F5-TTS architecture.
- The README calls out two main use cases: (a) fine-tuning the model on custom audio/transcription data (via `fine_tuning.sh`), and (b) synthesizing speech from text or reference audio (via `infer.sh` or the CLI/Gradio interfaces).

##### File structure
- Top-level:
- Dockerfile, shell scripts (`fine_tuning.sh`, `infer.sh`), Python entry scripts (`check_vocab_pretrained.py`, `convert_sr.py`, etc.), `pyproject.toml` for packaging.
- `data/`: raw audio and vocabulary files.
- `src/f5_tts/`: the core Python package, subdivided into:
– `configs/`: YAML files defining model hyperparameters (E2TTS vs. F5TTS variants).
– `model/`: modules implementing backbones (DIT, MMDIT, UNETT), datasets, the trainer, and utility functions.
– `train/`: dataset preparation scripts and CLI/Gradio interfaces for fine-tuning.
– `eval/`: evaluation scripts and utilities for objective metrics.
– `infer/`: inference logic (CLI, Gradio, speech editing) plus examples.
– `api.py`, `socket_server.py`/`socket_client.py`: programmatic and network-based serving.
– `scripts/`: helper scripts (count params, epochs, etc.).

##### Extract insights from the README
- High-level pipeline: Data prep → embedding expansion → feature extraction → fine-tuning → inference.
- External dependencies: PyTorch (+ torchaudio), Whisper for ASR, sox/ffmpeg for audio I/O.
- Architecturally, there’s a single package exposing both training and serving entry points, plus shell wrappers for reproducibility.

##### Components and relationships
a. Main components
- Data Storage (raw audio files, transcripts)
- Preprocessing Module
- Vocabulary Manager (check & extend embeddings)
- Feature Extractor (spectrogram, pitch, etc.)
- Model Trainer (F5-TTS backbone + cfm + optimizer)
- Model Repository (saved checkpoints)
- Evaluation Module
- Inference Service
- CLI Interface
- Gradio Web UI
- Socket Server / Client
- External Services & Tools (Whisper ASR, sox, ffmpeg)
b. Interactions and data flow
1. Raw data → Preprocessing scripts (`prepare_*`) → formatted CSV / metadata
2. Metadata & vocab → Vocabulary Manager → updated embedding matrices
3. Feature Extractor reads audio based on metadata → tensor inputs
4. Model Trainer consumes features + configs → trains → writes checkpoints to Model Repository
5. Evaluation Module pulls checkpoints → runs test scripts → metrics
6. Inference Service loads checkpoint → uses Whisper ASR (if needed) + feature modules → synthesizes waveform → outputs audio file or streams via socket
7. CLI/Gradio wrap the Inference Service for user interaction
c. Architectural patterns & design principles
- Modular package structure for clear separation of concerns (configs, model, training, inference, evaluation).
- Pipeline orchestration via shell scripts for reproducibility.
- Plugin-style backbones: easy to swap DIT/MM-DIT/UNETT.
- Dual serving modes: local CLI/Gradio vs. networked socket server.
d. Key technologies & libraries
- PyTorch & torchaudio for modeling and audio ops
- Whisper ASR (openAI/whisper) for optional reference-text extraction
- sox, ffmpeg for audio processing
- YAML for config management
- Gradio for web-based inference GUI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        DS["Data Storage\n(raw audio & transcripts)"]:::storage
    end

    %% Preprocessing
    subgraph "Preprocessing Module"
        PM1["prepare_metadata.py"]:::batch
        PM2["prepare_csv_wavs.py"]:::batch
        PM3["prepare_emilia.py"]:::batch
        PM4["prepare_libritts.py"]:::batch
        PM5["prepare_ljspeech.py"]:::batch
        PM6["prepare_wenetspeech4tts.py"]:::batch
    end

    %% Vocabulary Manager
    subgraph "Vocabulary Manager"
        VM1["check_vocab_pretrained.py"]:::batch
        VM2["extend_embedding_pretrained.py"]:::batch
        VM3["vocab.txt"]:::storage
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        C1["E2TTS_Base.yaml"]:::batch
        C2["E2TTS_Small.yaml"]:::batch
        C3["F5TTS_Base.yaml"]:::batch
        C4["F5TTS_Small.yaml"]:::batch
        C5["F5TTS_v1_Base.yaml"]:::batch
    end

    %% Feature Extraction
    subgraph "Feature Extraction"
        FE1["modules.py"]:::batch
        FE2["utils.py"]:::batch
    end

    %% Model Training
    subgraph "Model Training"
        MT1["trainer.py"]:::batch
        MT2["train.py"]:::batch
        FT["fine_tuning.sh"]:::batch
        subgraph "Backbone Implementations"
            BB1["dit.py"]:::batch
            BB2["mmdit.py"]:::batch
            BB3["unett.py"]:::batch
        end
    end

    %% Model Repository
    MR["Model Repository\n(checkpoints)"]:::storage

    %% Evaluation Module
    subgraph "Evaluation Module"
        E1["ecapa_tdnn.py"]:::batch
        E2["eval_infer_batch.py"]:::batch
        E3["eval_infer_batch.sh"]:::batch
        E4["eval_librispeech_test_clean.py"]:::batch
        E5["eval_seedtts_testset.py"]:::batch
        E6["eval_utmos.py"]:::batch
        E7["utils_eval.py"]:::batch
    end

    %% Inference Service
    subgraph "Inference Service"
        subgraph "CLI Interface"
            I1["infer_cli.py"]:::real
            I2["infer.sh"]:::real
        end
        subgraph "Gradio Web UI"
            I3["infer_gradio.py"]:::real
        end
        subgraph "Speech Editing"
            I4["speech_edit.py"]:::real
        end
        subgraph "Inference Utilities"
            I5["utils_infer.py"]:::real
        end
        subgraph "Examples"
            EX1["basic.toml"]:::batch
            EX2["basic_ref_en.wav"]:::batch
            EX3["basic_ref_zh.wav"]:::batch
        end
    end

    %% Serving
    subgraph "Programmatic & Network Serving"
        API["api.py"]:::real
        SS["socket_server.py"]:::real
        SC["socket_client.py"]:::real
    end

    %% Helpers & Packaging
    subgraph "Helper Scripts"
        H1["count_params_gflops.py"]:::batch
        H2["count_max_epoch.py"]:::batch
    end
    subgraph "Packaging & Entry Points"
        P1["pyproject.toml"]:::batch
        P2["check_vocab_pretrained.py"]:::batch
        P3["convert_sr.py"]:::batch
        P4["prepare_metadata.py"]:::batch
        P5["extend_embedding_pretrained.py"]:::batch
    end

    %% External Tools
    Whisper["Whisper ASR"]:::external
    SoxFfmpeg["sox/ffmpeg"]:::external

    %% Data Flow
    DS -->|provides raw data| PM1
    DS --> PM2
    DS --> PM3
    DS --> PM4
    DS --> PM5
    DS --> PM6

    PM1 --> VM1
    PM1 --> VM2
    VM1 --> VM3
    VM2 --> VM3

    VM3 --> FE1
    VM3 --> FE2
    Whisper -.-> FE2
    SoxFfmpeg -.-> FE2

    FE1 --> MT1
    FE2 --> MT1
    C1 --> MT1
    C2 --> MT1
    C3 --> MT1
    C4 --> MT1
    C5 --> MT1

    BB1 --> MT1
    BB2 --> MT1
    BB3 --> MT1
    FT --> MT2
    MT1 --> MR
    MT2 --> MR

    MR --> E1
    MR --> E2
    MR --> E4
    MR --> E5
    MR --> E6

    E2 --> E3
    E7 --> E1
    E7 --> E2

    MR --> I5
    I5 --> I1
    I5 --> I3
    I5 --> I4

    I1 --> API
    I3 --> API
    API --> SS
    SS --> SC

    H1 --> MT1
    H2 --> MT1

    P1 --> VM1
    P1 --> VM2
    P2 --> VM1
    P3 --> FE2
    P4 --> PM1
    P5 --> VM2

    %% Click Events
    click DS "https://github.com/nguyenthienhy/f5-tts-vietnamese/tree/main/data/"
    click PM1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/prepare_metadata.py"
    click PM2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/datasets/prepare_csv_wavs.py"
    click PM3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/datasets/prepare_emilia.py"
    click PM4 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/datasets/prepare_libritts.py"
    click PM5 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/datasets/prepare_ljspeech.py"
    click PM6 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/datasets/prepare_wenetspeech4tts.py"
    click VM1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/check_vocab_pretrained.py"
    click VM2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/extend_embedding_pretrained.py"
    click VM3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/data/Emilia_ZH_EN_pinyin/vocab.txt"
    click C1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/configs/E2TTS_Base.yaml"
    click C2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/configs/E2TTS_Small.yaml"
    click C3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/configs/F5TTS_Base.yaml"
    click C4 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/configs/F5TTS_Small.yaml"
    click C5 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/configs/F5TTS_v1_Base.yaml"
    click FE1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/modules.py"
    click FE2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/utils.py"
    click MT1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/trainer.py"
    click MT2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/train/train.py"
    click FT "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/fine_tuning.sh"
    click BB1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/backbones/dit.py"
    click BB2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/backbones/mmdit.py"
    click BB3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/model/backbones/unett.py"
    click E1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/ecapa_tdnn.py"
    click E2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/eval_infer_batch.py"
    click E3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/eval_infer_batch.sh"
    click E4 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/eval_librispeech_test_clean.py"
    click E5 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/eval_seedtts_testset.py"
    click E6 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/eval_utmos.py"
    click E7 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/eval/utils_eval.py"
    click I1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/infer_cli.py"
    click I2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/infer.sh"
    click I3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/infer_gradio.py"
    click I4 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/speech_edit.py"
    click I5 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/utils_infer.py"
    click EX1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/examples/basic/basic.toml"
    click EX2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/examples/basic/basic_ref_en.wav"
    click EX3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/infer/examples/basic/basic_ref_zh.wav"
    click API "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/api.py"
    click SS "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/socket_server.py"
    click SC "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/socket_client.py"
    click H1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/scripts/count_params_gflops.py"
    click H2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/src/f5_tts/scripts/count_max_epoch.py"
    click P1 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/pyproject.toml"
    click P2 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/check_vocab_pretrained.py"
    click P3 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/convert_sr.py"
    click P4 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/prepare_metadata.py"
    click P5 "https://github.com/nguyenthienhy/f5-tts-vietnamese/blob/main/extend_embedding_pretrained.py"

    %% Styles
    classDef storage fill:#f9f,stroke:#333,stroke-width:1px;
    classDef batch fill:#bbf,stroke:#333,stroke-width:1px;
    classDef real fill:#bfb,stroke:#333,stroke-width:1px;
    classDef external fill:#fcc,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    class DS,VM3,MR storage;
    class PM1,PM2,PM3,PM4,PM5,PM6,VM1,VM2,C1,C2,C3,C4,C5,FE1,FE2,MT1,MT2,FT,BB1,BB2,BB3,E1,E2,E3,E4,E5,E6,E7,H1,H2,P1,P2,P3,P4,P5 batch;
    class I1,I2,I3,I4,I5,API,SS,SC real;
    class Whisper,SoxFfmpeg external;
```
