# swivid/f5-tts architecture

> Architecture diagram of the GitHub repository swivid/f5-tts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swivid/f5-tts
- Repository: https://github.com/swivid/f5-tts

## Overview

#### Purpose and entry points
F5-TTS is a Python/PyTorch text-to-speech system that generates speech conditioned on reference audio/text using flow matching. It ships inference, training/finetuning, evaluation, and optimized serving paths. Public programmatic access is in `src/f5_tts/api.py`; packaged CLI/UI entry points are implemented by `src/f5_tts/infer/infer_cli.py`, `src/f5_tts/infer/infer_gradio.py`, and training counterparts under `src/f5_tts/train/`.

#### Model and generation core
The model layer separates conditional flow matching from selectable neural backbones:
- `src/f5_tts/model/cfm.py` implements the flow-matching generation/training wrapper and sampling path.
- `src/f5_tts/model/backbones/dit.py`, `src/f5_tts/model/backbones/mmdit.py`, and `src/f5_tts/model/backbones/unett.py` provide DiT, multimodal DiT, and Flat-UNet-style alternatives.
- `src/f5_tts/model/modules.py` contains shared model modules; model variants and dimensions are selected through YAMLs such as `src/f5_tts/configs/F5TTS_v1_Base.yaml` and `src/f5_tts/configs/E2TTS_Base.yaml`.

The generation boundary is mel/spectral model output to a vocoder, with Vocos and optional BigVGAN support (`src/third_party/BigVGAN`).

#### Local inference flow
Inference accepts reference audio plus its transcript (or can obtain transcription), target text, model/checkpoint selection, and sampling settings. It preprocesses/segments text and audio, runs conditional flow sampling, vocodes generated features, then concatenates or exports audio.
- Orchestration and user-facing flags/config loading: `src/f5_tts/infer/infer_cli.py`.
- Shared checkpoint loading, audio/text preprocessing, model/vocoder invocation, and chunked/multi-voice behavior: `src/f5_tts/infer/utils_infer.py`.
- Interactive Gradio workflows, including basic TTS, multi-speaker/style generation, and voice chat: `src/f5_tts/infer/infer_gradio.py`.
- Speech editing is a distinct inference application in `src/f5_tts/infer/speech_edit.py`.

This layer depends on downloadable pretrained assets, typically cached via Hugging Face.

#### Training and adaptation
Training consumes prepared speech/audio metadata, constructs datasets and model batches, and runs distributed-capable optimization through Hugging Face Accelerate.
- Dataset representation/loading: `src/f5_tts/model/dataset.py`.
- Training loop, checkpointing, and optimization coordination: `src/f5_tts/model/trainer.py` and `src/f5_tts/train/train.py`.
- Finetuning interfaces: `src/f5_tts/train/finetune_cli.py` and `src/f5_tts/train/finetune_gradio.py`.
- Corpus-specific preparation pipelines reside in `src/f5_tts/train/datasets/prepare_emilia.py` and related scripts.

Training configuration is model-family-specific YAML, not embedded in UI code.

#### Evaluation
Evaluation runs batch inference against benchmark datasets and computes intelligibility, speaker similarity, and quality-oriented metrics. Primary runners are `src/f5_tts/eval/eval_infer_batch.py`, `src/f5_tts/eval/eval_librispeech_test_clean.py`, and `src/f5_tts/eval/eval_seedtts_testset.py`; supporting integrations include ECAPA-TDNN and UTMOS. Evaluation is intentionally separated from the production inference path.

#### High-performance serving
An optional GPU deployment stack serves F5-TTS through NVIDIA Triton and TensorRT-LLM:
- Container/server orchestration: `src/f5_tts/runtime/triton_trtllm/Dockerfile.server`, `src/f5_tts/runtime/triton_trtllm/docker-compose.yml`, and `src/f5_tts/runtime/triton_trtllm/run.sh`.
- Triton model repository and Python backend: `src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py` and `src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt`.
- Checkpoint/vocoder conversion and export: `src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py` and `src/f5_tts/runtime/triton_trtllm/scripts/export_vocoder_to_onnx.py`.

Clients use HTTP or gRPC (`client_http.py`, `client_grpc.py`), making serving a clear boundary from local PyTorch inference.

#### Runtime and distribution
The project targets Python 3.10+ and PyTorch across CUDA, ROCm, Intel XPU/IPEX, and Apple Silicon; FFmpeg is required for audio handling. Docker packaging begins at `Dockerfile`; CI publishes Docker and PyPI artifacts via `.github/workflows/publish-docker-image.yaml` and `.github/workflows/publish-pypi.yaml`. Pretrained checkpoints are external rather than committed, with repository guidance in `ckpts/README.md`.

Main entry points: [`src/f5_tts/infer/utils_infer.py`](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/utils_infer.py) (Inference Utilities), [`src/f5_tts/model/trainer.py`](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/trainer.py) (Distributed Trainer), [`src/f5_tts/model/cfm.py`](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/cfm.py) (Conditional Flow Matching), [`ckpts/README.md`](https://github.com/swivid/f5-tts/blob/main/ckpts/README.md) (Pretrained Assets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_api["Python API<br/>programmatic interface<br/>[api.py]"]
  node_cli["Inference CLI<br/>local inference<br/>[infer_cli.py]"]
  node_gradio["Gradio UI<br/>interactive inference<br/>[infer_gradio.py]"]
  node_speech_edit["Speech Editing<br/>inference application<br/>[speech_edit.py]"]
end

subgraph group_core["Generation Core"]
  node_infer_utils["Inference Utilities<br/>orchestration<br/>[utils_infer.py]"]
  node_cfm["Conditional Flow Matching<br/>generation model<br/>[cfm.py]"]
  node_backbones["Neural Backbones<br/>[dit.py]"]
  node_model_modules["Shared Model Modules<br/>model components<br/>[modules.py]"]
  node_model_configs["Model Family YAMLs<br/>model configuration<br/>[F5TTS_v1_Base.yaml]"]
  node_vocoder["Vocoder<br/>spectrogram-to-audio"]
end

subgraph group_training["Training &amp; Evaluation"]
  node_dataset_prep["Corpus Preparation<br/>data pipeline<br/>[prepare_emilia.py]"]
  node_dataset["Speech Dataset<br/>dataset loader<br/>[dataset.py]"]
  node_trainer["Distributed Trainer<br/>optimization loop<br/>[trainer.py]"]
  node_finetune["Finetuning Interfaces<br/>CLI and UI<br/>[finetune_cli.py]"]
  node_evaluation["Benchmark Evaluation<br/>batch evaluation"]
end

subgraph group_serving["Optimized Serving"]
  node_triton_backend["Triton Python Backend<br/>GPU serving backend<br/>[model.py]"]
  node_triton_repo["Triton Model Repository<br/>serving configuration<br/>[config.pbtxt]"]
  node_conversion["TensorRT / ONNX Export<br/>deployment conversion"]
  node_triton_clients["HTTP / gRPC Clients<br/>remote clients<br/>[client_http.py]"]
end

node_pretrained[("Pretrained Assets<br/>external checkpoints<br/>[README.md]")]

node_api -->|"invokes"| node_infer_utils
node_cli -->|"orchestrates"| node_infer_utils
node_gradio -->|"runs TTS workflows"| node_infer_utils
node_speech_edit -->|"uses generation"| node_infer_utils
node_infer_utils -->|"loads"| node_pretrained
node_infer_utils -->|"samples mel features"| node_cfm
node_cfm -->|"selects denoising backbone"| node_backbones
node_backbones -->|"uses"| node_model_modules
node_model_configs -.->|"configures variant"| node_cfm
node_infer_utils -->|"vocodes features"| node_vocoder
node_dataset_prep -->|"prepares metadata"| node_dataset
node_dataset -->|"provides batches"| node_trainer
node_model_configs -.->|"configures training"| node_trainer
node_trainer -->|"optimizes"| node_cfm
node_finetune -->|"launches adaptation"| node_trainer
node_trainer -->|"writes checkpoints"| node_pretrained
node_evaluation -->|"runs batch inference"| node_infer_utils
node_conversion -->|"converts checkpoints"| node_pretrained
node_conversion -->|"exports deployment artifacts"| node_triton_repo
node_triton_repo -->|"loads"| node_triton_backend
node_triton_clients -->|"HTTP / gRPC requests"| node_triton_backend

click node_api "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/api.py"
click node_cli "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/infer_cli.py"
click node_gradio "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/infer_gradio.py"
click node_speech_edit "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/speech_edit.py"
click node_infer_utils "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/utils_infer.py"
click node_cfm "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/cfm.py"
click node_backbones "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/backbones/dit.py"
click node_model_modules "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/modules.py"
click node_model_configs "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/configs/F5TTS_v1_Base.yaml"
click node_vocoder "https://github.com/swivid/f5-tts/tree/main/src/third_party/BigVGAN"
click node_pretrained "https://github.com/swivid/f5-tts/blob/main/ckpts/README.md"
click node_dataset_prep "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/train/datasets/prepare_emilia.py"
click node_dataset "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/dataset.py"
click node_trainer "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/trainer.py"
click node_finetune "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/train/finetune_cli.py"
click node_evaluation "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/eval/eval_infer_batch.py"
click node_triton_backend "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py"
click node_triton_repo "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt"
click node_conversion "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py"
click node_triton_clients "https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/client_http.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_cli,node_gradio,node_speech_edit toneBlue
class node_infer_utils,node_cfm,node_backbones,node_model_modules,node_model_configs,node_vocoder toneAmber
class node_dataset_prep,node_dataset,node_trainer,node_finetune,node_evaluation toneMint
class node_triton_backend,node_triton_repo,node_conversion,node_triton_clients toneRose
class node_pretrained toneNeutral
```

## Components

### User Interfaces

- [Python API](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/api.py): programmatic interface · `src/f5_tts/api.py`
- [Inference CLI](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/infer_cli.py): local inference · `src/f5_tts/infer/infer_cli.py`
- [Gradio UI](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/infer_gradio.py): interactive inference · `src/f5_tts/infer/infer_gradio.py`
- [Speech Editing](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/speech_edit.py): inference application · `src/f5_tts/infer/speech_edit.py`

### Generation Core

- [Inference Utilities](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/infer/utils_infer.py): orchestration · Loads checkpoints and preprocesses, chunks, and vocodes requests. · `src/f5_tts/infer/utils_infer.py`
- [Conditional Flow Matching](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/cfm.py): generation model · Training wrapper and conditional sampling path. · `src/f5_tts/model/cfm.py`
- [Neural Backbones](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/backbones/dit.py): DiT / MMDiT / UNet · `src/f5_tts/model/backbones/dit.py`
- [Shared Model Modules](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/modules.py): model components · `src/f5_tts/model/modules.py`
- [Model Family YAMLs](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/configs/F5TTS_v1_Base.yaml): model configuration · `src/f5_tts/configs/F5TTS_v1_Base.yaml`
- [Vocoder](https://github.com/swivid/f5-tts/tree/main/src/third_party/BigVGAN): spectrogram-to-audio · Uses Vocos, with optional BigVGAN support. · `src/third_party/BigVGAN`

### Training & Evaluation

- [Corpus Preparation](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/train/datasets/prepare_emilia.py): data pipeline · `src/f5_tts/train/datasets/prepare_emilia.py`
- [Speech Dataset](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/dataset.py): dataset loader · `src/f5_tts/model/dataset.py`
- [Distributed Trainer](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/model/trainer.py): optimization loop · Accelerate-capable training and checkpointing. · `src/f5_tts/model/trainer.py`
- [Finetuning Interfaces](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/train/finetune_cli.py): CLI and UI · `src/f5_tts/train/finetune_cli.py`
- [Benchmark Evaluation](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/eval/eval_infer_batch.py): batch evaluation · Separately runs inference and intelligibility, similarity, and quality metrics. · `src/f5_tts/eval/eval_infer_batch.py`

### Optimized Serving

- [Triton Python Backend](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py): GPU serving backend · `src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py`
- [Triton Model Repository](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt): serving configuration · `src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt`
- [TensorRT / ONNX Export](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py): deployment conversion · Converts checkpoints and exports the vocoder. · `src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py`
- [HTTP / gRPC Clients](https://github.com/swivid/f5-tts/blob/main/src/f5_tts/runtime/triton_trtllm/client_http.py): remote clients · `src/f5_tts/runtime/triton_trtllm/client_http.py`

### Other components

- [Pretrained Assets](https://github.com/swivid/f5-tts/blob/main/ckpts/README.md): external checkpoints · Downloadable assets are typically cached through Hugging Face. · `ckpts/README.md`

## Connections

- Python API → Inference Utilities: invokes
- Inference CLI → Inference Utilities: orchestrates
- Gradio UI → Inference Utilities: runs TTS workflows
- Speech Editing → Inference Utilities: uses generation
- Inference Utilities → Pretrained Assets: loads
- Inference Utilities → Conditional Flow Matching: samples mel features
- Conditional Flow Matching → Neural Backbones: selects denoising backbone
- Neural Backbones → Shared Model Modules: uses
- Model Family YAMLs → Conditional Flow Matching: configures variant
- Inference Utilities → Vocoder: vocodes features
- Corpus Preparation → Speech Dataset: prepares metadata
- Speech Dataset → Distributed Trainer: provides batches
- Model Family YAMLs → Distributed Trainer: configures training
- Distributed Trainer → Conditional Flow Matching: optimizes
- Finetuning Interfaces → Distributed Trainer: launches adaptation
- Distributed Trainer → Pretrained Assets: writes checkpoints
- Benchmark Evaluation → Inference Utilities: runs batch inference
- TensorRT / ONNX Export → Pretrained Assets: converts checkpoints
- TensorRT / ONNX Export → Triton Model Repository: exports deployment artifacts
- Triton Model Repository → Triton Python Backend: loads
- HTTP / gRPC Clients → Triton Python Backend: HTTP / gRPC requests
