How swivid/f5-tts works
Purpose and entry points
F5-TTS is a Python/PyTorch text-to-speech system that generates speech conditioned on reference audio/text using flow matching. It ships inference, training/finetuning, evaluation, and optimized serving paths. Public programmatic access is in src/f5_tts/api.py; packaged CLI/UI entry points are implemented by src/f5_tts/infer/infer_cli.py, src/f5_tts/infer/infer_gradio.py, and training counterparts under src/f5_tts/train/.
Model and generation core
The model layer separates conditional flow matching from selectable neural backbones:
• src/f5_tts/model/cfm.py implements the flow-matching generation/training wrapper and sampling path.
• src/f5_tts/model/backbones/dit.py, src/f5_tts/model/backbones/mmdit.py, and src/f5_tts/model/backbones/unett.py provide DiT, multimodal DiT, and Flat-UNet-style alternatives.
• src/f5_tts/model/modules.py contains shared model modules; model variants and dimensions are selected through YAMLs such as src/f5_tts/configs/F5TTS_v1_Base.yaml and src/f5_tts/configs/E2TTS_Base.yaml.
The generation boundary is mel/spectral model output to a vocoder, with Vocos and optional BigVGAN support (src/third_party/BigVGAN).
Local inference flow
Inference accepts reference audio plus its transcript (or can obtain transcription), target text, model/checkpoint selection, and sampling settings. It preprocesses/segments text and audio, runs conditional flow sampling, vocodes generated features, then concatenates or exports audio.
• Orchestration and user-facing flags/config loading: src/f5_tts/infer/infer_cli.py.
• Shared checkpoint loading, audio/text preprocessing, model/vocoder invocation, and chunked/multi-voice behavior: src/f5_tts/infer/utils_infer.py.
• Interactive Gradio workflows, including basic TTS, multi-speaker/style generation, and voice chat: src/f5_tts/infer/infer_gradio.py.
• Speech editing is a distinct inference application in src/f5_tts/infer/speech_edit.py.
This layer depends on downloadable pretrained assets, typically cached via Hugging Face.
Training and adaptation
Training consumes prepared speech/audio metadata, constructs datasets and model batches, and runs distributed-capable optimization through Hugging Face Accelerate.
• Dataset representation/loading: src/f5_tts/model/dataset.py.
• Training loop, checkpointing, and optimization coordination: src/f5_tts/model/trainer.py and src/f5_tts/train/train.py.
• Finetuning interfaces: src/f5_tts/train/finetune_cli.py and src/f5_tts/train/finetune_gradio.py.
• Corpus-specific preparation pipelines reside in src/f5_tts/train/datasets/prepare_emilia.py and related scripts.
Training configuration is model-family-specific YAML, not embedded in UI code.
Evaluation
Evaluation runs batch inference against benchmark datasets and computes intelligibility, speaker similarity, and quality-oriented metrics. Primary runners are src/f5_tts/eval/eval_infer_batch.py, src/f5_tts/eval/eval_librispeech_test_clean.py, and src/f5_tts/eval/eval_seedtts_testset.py; supporting integrations include ECAPA-TDNN and UTMOS. Evaluation is intentionally separated from the production inference path.
High-performance serving
An optional GPU deployment stack serves F5-TTS through NVIDIA Triton and TensorRT-LLM:
• Container/server orchestration: src/f5_tts/runtime/triton_trtllm/Dockerfile.server, src/f5_tts/runtime/triton_trtllm/docker-compose.yml, and src/f5_tts/runtime/triton_trtllm/run.sh.
• Triton model repository and Python backend: src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py and src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt.
• Checkpoint/vocoder conversion and export: src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py and src/f5_tts/runtime/triton_trtllm/scripts/export_vocoder_to_onnx.py.
Clients use HTTP or gRPC (client_http.py, client_grpc.py), making serving a clear boundary from local PyTorch inference.
Runtime and distribution
The project targets Python 3.10+ and PyTorch across CUDA, ROCm, Intel XPU/IPEX, and Apple Silicon; FFmpeg is required for audio handling. Docker packaging begins at Dockerfile; CI publishes Docker and PyPI artifacts via .github/workflows/publish-docker-image.yaml and .github/workflows/publish-pypi.yaml. Pretrained checkpoints are external rather than committed, with repository guidance in ckpts/README.md.
Main entry points: utils_infer.py Inference Utilities · trainer.py Distributed Trainer · cfm.py Conditional Flow Matching · README.md Pretrained Assets
Components 20
User Interfaces
- Python API
src/f5_tts/api.py - Inference CLI
src/f5_tts/infer/infer_cli.py - Gradio UI
src/f5_tts/infer/infer_gradio.py - Speech Editing
src/f5_tts/infer/speech_edit.py
Generation Core
- Inference Utilities · Loads checkpoints and preprocesses, chunks, and vocodes requests.
src/f5_tts/infer/utils_infer.py - Conditional Flow Matching · Training wrapper and conditional sampling path.
src/f5_tts/model/cfm.py - Neural Backbones
src/f5_tts/model/backbones/dit.py - Shared Model Modules
src/f5_tts/model/modules.py - Model Family YAMLs
src/f5_tts/configs/F5TTS_v1_Base.yaml - Vocoder · Uses Vocos, with optional BigVGAN support.
src/third_party/BigVGAN
Training & Evaluation
- Corpus Preparation
src/f5_tts/train/datasets/prepare_emilia.py - Speech Dataset
src/f5_tts/model/dataset.py - Distributed Trainer · Accelerate-capable training and checkpointing.
src/f5_tts/model/trainer.py - Finetuning Interfaces
src/f5_tts/train/finetune_cli.py - Benchmark Evaluation · Separately runs inference and intelligibility, similarity, and quality metrics.
src/f5_tts/eval/eval_infer_batch.py
Optimized Serving
- Triton Python Backend
src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/1/model.py - Triton Model Repository
src/f5_tts/runtime/triton_trtllm/model_repo_f5_tts/f5_tts/config.pbtxt - TensorRT / ONNX Export · Converts checkpoints and exports the vocoder.
src/f5_tts/runtime/triton_trtllm/scripts/convert_checkpoint.py - HTTP / gRPC Clients
src/f5_tts/runtime/triton_trtllm/client_http.py
Other components
- Pretrained Assets · Downloadable assets are typically cached through Hugging Face.
ckpts/README.md
Connections 21
- Python API → Inference Utilities: invokes
- Inference CLI → Inference Utilities: orchestrates
- Gradio UI → Inference Utilities: runs TTS workflows
- Speech Editing → Inference Utilities: uses generation
- Inference Utilities → Pretrained Assets: loads
- Inference Utilities → Conditional Flow Matching: samples mel features
- Conditional Flow Matching → Neural Backbones: selects denoising backbone
- Neural Backbones → Shared Model Modules: uses
- Model Family YAMLs → Conditional Flow Matching: configures variant
- Inference Utilities → Vocoder: vocodes features
- Corpus Preparation → Speech Dataset: prepares metadata
- Speech Dataset → Distributed Trainer: provides batches
- Model Family YAMLs → Distributed Trainer: configures training
- Distributed Trainer → Conditional Flow Matching: optimizes
- Finetuning Interfaces → Distributed Trainer: launches adaptation
- Distributed Trainer → Pretrained Assets: writes checkpoints
- Benchmark Evaluation → Inference Utilities: runs batch inference
- TensorRT / ONNX Export → Pretrained Assets: converts checkpoints
- TensorRT / ONNX Export → Triton Model Repository: exports deployment artifacts
- Triton Model Repository → Triton Python Backend: loads
- HTTP / gRPC Clients → Triton Python Backend: HTTP / gRPC requests