How atlonxp/f5-tts works
Project type and purpose
• F5-TTS is a full-stack TTS framework. It provides:
– A Python library (src/f5_tts/…) for training, inference, evaluation.
– Command-line and Gradio-based front ends.
– A runtime deployment via Triton + TensorRT-LLM.
– A Django-based REST API (in api/…) for serving/storing inference results.
Main components
a. Clients & Interfaces
• CLI client (f5-tts_infer-cli, socket_client.py)
• Gradio web UI (f5-tts_infer-gradio, infer_gradio.py)
• Django Admin/API front end (api/)
b. Core Library
• src/f5_tts/… – model definitions (CFM, ConvNeXtV2, UNet, DiT), dataset loaders, trainer, inference utilities, scripts.
• Third-party modules (src/third_party/deep_phonemizer, BigVGAN, ai4thai)
c. Model Runtime
• PyTorch inference pipeline (CPU/GPU)
• Triton + TensorRT-LLM server (src/f5_tts/runtime/triton_trtllm/…) exposing gRPC/HTTP endpoints
• Socket-server mode (socket_server.py)
d. Data & Storage
• SQLite (api/db.sqlite3) – metadata, usage logs
• Media storage (api/media/…) – uploaded/generated WAVs
• Static assets (api/staticfiles/…) for Django docs & swagger UI
• Model checkpoints (ckpts/), dataset files (data/), demo files (demo/)
e. External Services
• Hugging Face Hub (model hosting)
• GitHub Container Registry (Docker image)
• Pre-commit CI, GitHub Actions
Interactions & data flow
1. User issues a TTS request via:
• CLI → directly calls src/f5_tts/infer/utils_infer.py or via socket server
• Gradio UI → infer_gradio.py → same core library
• HTTP to Django API → Django views (api/tts/api/tts.py) → invokes core library → stores result to SQLite & media folder
• gRPC/HTTP to Triton server → Triton loads serialized models (model_repo_f5_tts) → returns mel-spectrogram/audio bytes
2. Core library routes calls:
• To PyTorch modules (model/backbones, cfm, modules, vocoder)
• To third-party phonemizer or vocoder (BigVGAN) as needed
3. Training & Evaluation
• Scripts under src/f5_tts/train/… call dataset preparers → trainer → save checkpoints
• Evaluation modules under src/f5_tts/eval/… load models & datasets → compute metrics
Architectural patterns & design principles
• Modular package separation: library vs. interface vs. runtime vs. API
• Adapter pattern for different inference backends (direct PyTorch vs. socket vs. Triton)
• MVC-style for Django API: models (usagelog, speaker, etc.) → views (API endpoints) → templates/static for docs
• Configuration-as-code with YAML (in configs/) and TOML (in infer/examples/)
• All major components clearly labeled
• On staticfiles for API documentation (Swagger/Redoc)