How lm-sys/fastchat works
Purpose and Entry Points
FastChat is a Python platform for the full LLM-chatbot lifecycle: conversation-data preparation, fine-tuning, local/distributed inference, web/API serving, and benchmark/Arena evaluation. Primary executable modules are the interactive CLI at fastchat/serve/cli.py, distributed controller at fastchat/serve/controller.py, local model worker at fastchat/serve/model_worker.py, Gradio UIs at fastchat/serve/gradio_web_server.py and fastchat/serve/gradio_web_server_multi.py, and OpenAI-compatible server at fastchat/serve/openai_api_server.py.
Conversation and Model Abstraction
Conversation templates and common constants normalize prompts across model families in fastchat/conversation.py and fastchat/constants.py. Model selection is centralized through adapters and registry logic in fastchat/model/model_adapter.py and fastchat/model/model_registry.py; family-specific implementations cover nonstandard architectures. This boundary lets serving, CLI, and training flows accept model paths/repository IDs while applying model-appropriate tokenization, prompt formatting, and loading behavior.
Inference and Distributed Serving Plane
The serving control plane separates routing from GPU-backed execution. fastchat/serve/controller.py maintains worker registration and dispatches requests; workers inherit common service behavior from fastchat/serve/base_model_worker.py. The Transformers worker in fastchat/serve/model_worker.py hosts models and streams generation, while specialized workers support high-throughput or alternate runtimes such as fastchat/serve/vllm_worker.py, fastchat/serve/sglang_worker.py, fastchat/serve/lightllm_worker.py, fastchat/serve/mlx_worker.py, and fastchat/serve/dashinfer_worker.py. Workers can scale horizontally and register multiple replicas/model types with a controller.
Client Interfaces and Protocol Boundary
Human-facing Gradio interfaces are implemented in fastchat/serve/gradio_web_server.py; multi-model and side-by-side Arena variants reside in fastchat/serve/gradio_web_server_multi.py and fastchat/serve/gradio_block_arena_anony.py. REST compatibility is exposed by fastchat/serve/openai_api_server.py, with request/response schemas in fastchat/protocol/openai_api_protocol.py. The UI/API tier calls the controller or configured providers rather than owning model execution.
External Model Providers
fastchat/serve/api_provider.py provides an adapter boundary for remotely hosted models, including OpenAI-style, Anthropic, Gemini, and other APIs configured by endpoint metadata. This allows Arena and multi-model UI flows to combine remote API models with controller-registered local workers. Hugging Face-compatible generation serving is separately exposed through fastchat/serve/huggingface_api.py; model artifacts are commonly obtained from Hugging Face repositories.
Training and Data Pipeline
Training scripts consume conversation JSON and fine-tune Hugging Face/PyTorch models, with standard, memory-oriented, LoRA, and architecture-specific paths in fastchat/train/train.py, fastchat/train/train_mem.py, and fastchat/train/train_lora.py. Conversation preparation is a file-oriented transformation pipeline under fastchat/data/, notably fastchat/data/clean_sharegpt.py, fastchat/data/split_long_conversation.py, and fastchat/data/prepare_all.py. Training supports distributed PyTorch execution, FSDP, GPU acceleration, and optional experiment tracking integrations.
Evaluation, Arena Analytics, and Outputs
MT-Bench evaluation generates model/API answers, asks strong external LLMs to judge them, and aggregates results through fastchat/llm_judge/gen_model_answer.py, fastchat/llm_judge/gen_api_answer.py, fastchat/llm_judge/gen_judgment.py, and fastchat/llm_judge/show_result.py. Production Arena conversation/vote processing, deduplication, rating calculations, and leaderboard reporting live under fastchat/serve/monitor/, especially fastchat/serve/monitor/clean_battle_data.py, fastchat/serve/monitor/elo_analysis.py, and fastchat/serve/monitor/monitor.py. Docker deployment assets are provided in docker/Dockerfile and docker/docker-compose.yml.
Main entry points: controller.py Controller · model_adapter.py Model adapters & registry · model_worker.py Transformers worker · gradio_web_server_multi.py Multi-model Arena UI
Components 19
Serving plane
Routes requests to local or remote model backends.
- Controller · Registers workers and dispatches generation requests.
fastchat/serve/controller.py - Base model worker · Shared worker registration and service behavior.
fastchat/serve/base_model_worker.py - Transformers worker · Loads local models and streams generation.
fastchat/serve/model_worker.py - Alternate runtime workers · vLLM, SGLang, LightLLM, MLX, and DashInfer backends.
fastchat/serve/vllm_worker.py - Remote provider adapter · Routes configured models to hosted LLM APIs.
fastchat/serve/api_provider.py - Hugging Face generation API
fastchat/serve/huggingface_api.py
Client interfaces
- Interactive CLI
fastchat/serve/cli.py - Gradio UI
fastchat/serve/gradio_web_server.py - Multi-model Arena UI · Provides side-by-side and anonymous Arena interactions.
fastchat/serve/gradio_web_server_multi.py - OpenAI-compatible API
fastchat/serve/openai_api_server.py - OpenAI schemas
fastchat/protocol/openai_api_protocol.py
Data and training
- Conversation data preparation · Cleans, splits, and prepares ShareGPT-style conversation JSON.
fastchat/data/prepare_all.py - Fine-tuning · Standard, memory-oriented, and LoRA PyTorch fine-tuning.
fastchat/train/train.py
Evaluation and Arena
- MT-Bench answer generation · Generates answers from local models or APIs.
fastchat/llm_judge/gen_model_answer.py - MT-Bench judging & results · Uses external LLM judges and aggregates benchmark results.
fastchat/llm_judge/gen_judgment.py - Arena analytics · Cleans battles, computes ratings, and publishes leaderboard views.
fastchat/serve/monitor/elo_analysis.py
Other components
- Conversation templates · Normalizes prompts and shared chat constants.
fastchat/conversation.py - Model adapters & registry · Selects family-specific loading, tokenization, and formatting behavior.
fastchat/model/model_adapter.py - Docker deployment
docker/docker-compose.yml
Connections 22
- Interactive CLI → Conversation templates: formats prompts
- Interactive CLI → Model adapters & registry: loads model
- Gradio UI → Controller: submits generation
- Multi-model Arena UI → Controller: uses local models
- Multi-model Arena UI → Remote provider adapter: uses remote models
- OpenAI-compatible API → OpenAI schemas: uses schemas
- OpenAI-compatible API → Controller: routes requests
- Controller → Base model worker: registers and dispatches
- Base model worker → Transformers worker: shared base
- Base model worker → Alternate runtime workers: shared base
- Transformers worker → Model adapters & registry: loads through
- Transformers worker → Conversation templates: formats prompts
- Alternate runtime workers → Model adapters & registry: selects model behavior
- Hugging Face generation API → Model adapters & registry: loads models
- Conversation data preparation → Fine-tuning: prepared conversation JSON
- Fine-tuning → Model adapters & registry: model family behavior
- MT-Bench answer generation → Controller: queries local models
- MT-Bench answer generation → Remote provider adapter: queries APIs
- MT-Bench answer generation → MT-Bench judging & results: generated answers
- Multi-model Arena UI → Arena analytics: conversation and vote logs
- Docker deployment → Controller: deploys
- Docker deployment → Transformers worker: deploys