# lm-sys/fastchat architecture

> Architecture diagram of the GitHub repository lm-sys/fastchat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lm-sys/fastchat
- Repository: https://github.com/lm-sys/fastchat

## Overview

#### Purpose and Entry Points
FastChat is a Python platform for the full LLM-chatbot lifecycle: conversation-data preparation, fine-tuning, local/distributed inference, web/API serving, and benchmark/Arena evaluation. Primary executable modules are the interactive CLI at `fastchat/serve/cli.py`, distributed controller at `fastchat/serve/controller.py`, local model worker at `fastchat/serve/model_worker.py`, Gradio UIs at `fastchat/serve/gradio_web_server.py` and `fastchat/serve/gradio_web_server_multi.py`, and OpenAI-compatible server at `fastchat/serve/openai_api_server.py`.

#### Conversation and Model Abstraction
Conversation templates and common constants normalize prompts across model families in `fastchat/conversation.py` and `fastchat/constants.py`. Model selection is centralized through adapters and registry logic in `fastchat/model/model_adapter.py` and `fastchat/model/model_registry.py`; family-specific implementations cover nonstandard architectures. This boundary lets serving, CLI, and training flows accept model paths/repository IDs while applying model-appropriate tokenization, prompt formatting, and loading behavior.

#### Inference and Distributed Serving Plane
The serving control plane separates routing from GPU-backed execution. `fastchat/serve/controller.py` maintains worker registration and dispatches requests; workers inherit common service behavior from `fastchat/serve/base_model_worker.py`. The Transformers worker in `fastchat/serve/model_worker.py` hosts models and streams generation, while specialized workers support high-throughput or alternate runtimes such as `fastchat/serve/vllm_worker.py`, `fastchat/serve/sglang_worker.py`, `fastchat/serve/lightllm_worker.py`, `fastchat/serve/mlx_worker.py`, and `fastchat/serve/dashinfer_worker.py`. Workers can scale horizontally and register multiple replicas/model types with a controller.

#### Client Interfaces and Protocol Boundary
Human-facing Gradio interfaces are implemented in `fastchat/serve/gradio_web_server.py`; multi-model and side-by-side Arena variants reside in `fastchat/serve/gradio_web_server_multi.py` and `fastchat/serve/gradio_block_arena_anony.py`. REST compatibility is exposed by `fastchat/serve/openai_api_server.py`, with request/response schemas in `fastchat/protocol/openai_api_protocol.py`. The UI/API tier calls the controller or configured providers rather than owning model execution.

#### External Model Providers
`fastchat/serve/api_provider.py` provides an adapter boundary for remotely hosted models, including OpenAI-style, Anthropic, Gemini, and other APIs configured by endpoint metadata. This allows Arena and multi-model UI flows to combine remote API models with controller-registered local workers. Hugging Face-compatible generation serving is separately exposed through `fastchat/serve/huggingface_api.py`; model artifacts are commonly obtained from Hugging Face repositories.

#### Training and Data Pipeline
Training scripts consume conversation JSON and fine-tune Hugging Face/PyTorch models, with standard, memory-oriented, LoRA, and architecture-specific paths in `fastchat/train/train.py`, `fastchat/train/train_mem.py`, and `fastchat/train/train_lora.py`. Conversation preparation is a file-oriented transformation pipeline under `fastchat/data/`, notably `fastchat/data/clean_sharegpt.py`, `fastchat/data/split_long_conversation.py`, and `fastchat/data/prepare_all.py`. Training supports distributed PyTorch execution, FSDP, GPU acceleration, and optional experiment tracking integrations.

#### Evaluation, Arena Analytics, and Outputs
MT-Bench evaluation generates model/API answers, asks strong external LLMs to judge them, and aggregates results through `fastchat/llm_judge/gen_model_answer.py`, `fastchat/llm_judge/gen_api_answer.py`, `fastchat/llm_judge/gen_judgment.py`, and `fastchat/llm_judge/show_result.py`. Production Arena conversation/vote processing, deduplication, rating calculations, and leaderboard reporting live under `fastchat/serve/monitor/`, especially `fastchat/serve/monitor/clean_battle_data.py`, `fastchat/serve/monitor/elo_analysis.py`, and `fastchat/serve/monitor/monitor.py`. Docker deployment assets are provided in `docker/Dockerfile` and `docker/docker-compose.yml`.

Main entry points: [`fastchat/serve/controller.py`](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/controller.py) (Controller), [`fastchat/model/model_adapter.py`](https://github.com/lm-sys/fastchat/blob/main/fastchat/model/model_adapter.py) (Model adapters & registry), [`fastchat/serve/model_worker.py`](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/model_worker.py) (Transformers worker), [`fastchat/serve/gradio_web_server_multi.py`](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/gradio_web_server_multi.py) (Multi-model Arena UI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving plane"]
  node_controller{{"Controller<br/>routing control plane<br/>[controller.py]"}}
  node_base_worker["Base model worker<br/>worker foundation"]
  node_transformers_worker["Transformers worker<br/>GPU inference worker<br/>[model_worker.py]"]
  node_alternate_workers["Alternate runtime workers<br/>inference workers<br/>[vllm_worker.py]"]
  node_provider_adapter{{"Remote provider adapter<br/>provider boundary<br/>[api_provider.py]"}}
  node_hf_api["Hugging Face generation API<br/>generation server<br/>[huggingface_api.py]"]
end

subgraph group_clients["Client interfaces"]
  node_cli["Interactive CLI<br/>CLI entry point<br/>[cli.py]"]
  node_gradio_ui["Gradio UI<br/>web UI"]
  node_arena_ui["Multi-model Arena UI<br/>web UI"]
  node_openai_api["OpenAI-compatible API<br/>REST server"]
  node_openai_protocol["OpenAI schemas<br/>protocol schemas"]
end

subgraph group_training["Data and training"]
  node_data_prep["Conversation data preparation<br/>data pipeline<br/>[prepare_all.py]"]
  node_training["Fine-tuning<br/>training pipeline<br/>[train.py]"]
end

subgraph group_evaluation["Evaluation and Arena"]
  node_mtbench_answers["MT-Bench answer generation<br/>evaluation pipeline"]
  node_mtbench_judging["MT-Bench judging &amp; results<br/>evaluation pipeline<br/>[gen_judgment.py]"]
  node_arena_analytics["Arena analytics<br/>analytics pipeline<br/>[elo_analysis.py]"]
end

node_conversation["Conversation templates<br/>prompt abstraction<br/>[conversation.py]"]
node_model_adapters{{"Model adapters &amp; registry<br/>model abstraction<br/>[model_adapter.py]"}}
node_docker["Docker deployment<br/>deployment assets<br/>[docker-compose.yml]"]

node_cli -->|"formats prompts"| node_conversation
node_cli -->|"loads model"| node_model_adapters
node_gradio_ui -->|"submits generation"| node_controller
node_arena_ui -->|"uses local models"| node_controller
node_arena_ui -->|"uses remote models"| node_provider_adapter
node_openai_api -->|"uses schemas"| node_openai_protocol
node_openai_api -->|"routes requests"| node_controller
node_controller -->|"registers and dispatches"| node_base_worker
node_base_worker -->|"shared base"| node_transformers_worker
node_base_worker -->|"shared base"| node_alternate_workers
node_transformers_worker -->|"loads through"| node_model_adapters
node_transformers_worker -->|"formats prompts"| node_conversation
node_alternate_workers -->|"selects model behavior"| node_model_adapters
node_hf_api -->|"loads models"| node_model_adapters
node_data_prep -->|"prepared conversation JSON"| node_training
node_training -->|"model family behavior"| node_model_adapters
node_mtbench_answers -->|"queries local models"| node_controller
node_mtbench_answers -->|"queries APIs"| node_provider_adapter
node_mtbench_answers -->|"generated answers"| node_mtbench_judging
node_arena_ui -->|"conversation and vote logs"| node_arena_analytics
node_docker -.->|"deploys"| node_controller
node_docker -.->|"deploys"| node_transformers_worker

click node_conversation "https://github.com/lm-sys/fastchat/blob/main/fastchat/conversation.py"
click node_model_adapters "https://github.com/lm-sys/fastchat/blob/main/fastchat/model/model_adapter.py"
click node_cli "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/cli.py"
click node_gradio_ui "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/gradio_web_server.py"
click node_arena_ui "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/gradio_web_server_multi.py"
click node_openai_api "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/openai_api_server.py"
click node_openai_protocol "https://github.com/lm-sys/fastchat/blob/main/fastchat/protocol/openai_api_protocol.py"
click node_controller "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/controller.py"
click node_base_worker "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/base_model_worker.py"
click node_transformers_worker "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/model_worker.py"
click node_alternate_workers "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/vllm_worker.py"
click node_provider_adapter "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/api_provider.py"
click node_hf_api "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/huggingface_api.py"
click node_data_prep "https://github.com/lm-sys/fastchat/blob/main/fastchat/data/prepare_all.py"
click node_training "https://github.com/lm-sys/fastchat/blob/main/fastchat/train/train.py"
click node_mtbench_answers "https://github.com/lm-sys/fastchat/blob/main/fastchat/llm_judge/gen_model_answer.py"
click node_mtbench_judging "https://github.com/lm-sys/fastchat/blob/main/fastchat/llm_judge/gen_judgment.py"
click node_arena_analytics "https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/monitor/elo_analysis.py"
click node_docker "https://github.com/lm-sys/fastchat/blob/main/docker/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_controller,node_base_worker,node_transformers_worker,node_alternate_workers,node_provider_adapter,node_hf_api toneBlue
class node_cli,node_gradio_ui,node_arena_ui,node_openai_api,node_openai_protocol toneAmber
class node_data_prep,node_training toneMint
class node_mtbench_answers,node_mtbench_judging,node_arena_analytics toneRose
class node_conversation,node_model_adapters,node_docker toneNeutral
```

## Components

### Serving plane

Routes requests to local or remote model backends.

- [Controller](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/controller.py): routing control plane · Registers workers and dispatches generation requests. · `fastchat/serve/controller.py`
- [Base model worker](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/base_model_worker.py): worker foundation · Shared worker registration and service behavior. · `fastchat/serve/base_model_worker.py`
- [Transformers worker](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/model_worker.py): GPU inference worker · Loads local models and streams generation. · `fastchat/serve/model_worker.py`
- [Alternate runtime workers](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/vllm_worker.py): inference workers · vLLM, SGLang, LightLLM, MLX, and DashInfer backends. · `fastchat/serve/vllm_worker.py`
- [Remote provider adapter](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/api_provider.py): provider boundary · Routes configured models to hosted LLM APIs. · `fastchat/serve/api_provider.py`
- [Hugging Face generation API](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/huggingface_api.py): generation server · `fastchat/serve/huggingface_api.py`

### Client interfaces

- [Interactive CLI](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/cli.py): CLI entry point · `fastchat/serve/cli.py`
- [Gradio UI](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/gradio_web_server.py): web UI · `fastchat/serve/gradio_web_server.py`
- [Multi-model Arena UI](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/gradio_web_server_multi.py): web UI · Provides side-by-side and anonymous Arena interactions. · `fastchat/serve/gradio_web_server_multi.py`
- [OpenAI-compatible API](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/openai_api_server.py): REST server · `fastchat/serve/openai_api_server.py`
- [OpenAI schemas](https://github.com/lm-sys/fastchat/blob/main/fastchat/protocol/openai_api_protocol.py): protocol schemas · `fastchat/protocol/openai_api_protocol.py`

### Data and training

- [Conversation data preparation](https://github.com/lm-sys/fastchat/blob/main/fastchat/data/prepare_all.py): data pipeline · Cleans, splits, and prepares ShareGPT-style conversation JSON. · `fastchat/data/prepare_all.py`
- [Fine-tuning](https://github.com/lm-sys/fastchat/blob/main/fastchat/train/train.py): training pipeline · Standard, memory-oriented, and LoRA PyTorch fine-tuning. · `fastchat/train/train.py`

### Evaluation and Arena

- [MT-Bench answer generation](https://github.com/lm-sys/fastchat/blob/main/fastchat/llm_judge/gen_model_answer.py): evaluation pipeline · Generates answers from local models or APIs. · `fastchat/llm_judge/gen_model_answer.py`
- [MT-Bench judging & results](https://github.com/lm-sys/fastchat/blob/main/fastchat/llm_judge/gen_judgment.py): evaluation pipeline · Uses external LLM judges and aggregates benchmark results. · `fastchat/llm_judge/gen_judgment.py`
- [Arena analytics](https://github.com/lm-sys/fastchat/blob/main/fastchat/serve/monitor/elo_analysis.py): analytics pipeline · Cleans battles, computes ratings, and publishes leaderboard views. · `fastchat/serve/monitor/elo_analysis.py`

### Other components

- [Conversation templates](https://github.com/lm-sys/fastchat/blob/main/fastchat/conversation.py): prompt abstraction · Normalizes prompts and shared chat constants. · `fastchat/conversation.py`
- [Model adapters & registry](https://github.com/lm-sys/fastchat/blob/main/fastchat/model/model_adapter.py): model abstraction · Selects family-specific loading, tokenization, and formatting behavior. · `fastchat/model/model_adapter.py`
- [Docker deployment](https://github.com/lm-sys/fastchat/blob/main/docker/docker-compose.yml): deployment assets · `docker/docker-compose.yml`

## Connections

- Interactive CLI → Conversation templates: formats prompts
- Interactive CLI → Model adapters & registry: loads model
- Gradio UI → Controller: submits generation
- Multi-model Arena UI → Controller: uses local models
- Multi-model Arena UI → Remote provider adapter: uses remote models
- OpenAI-compatible API → OpenAI schemas: uses schemas
- OpenAI-compatible API → Controller: routes requests
- Controller → Base model worker: registers and dispatches
- Base model worker → Transformers worker: shared base
- Base model worker → Alternate runtime workers: shared base
- Transformers worker → Model adapters & registry: loads through
- Transformers worker → Conversation templates: formats prompts
- Alternate runtime workers → Model adapters & registry: selects model behavior
- Hugging Face generation API → Model adapters & registry: loads models
- Conversation data preparation → Fine-tuning: prepared conversation JSON
- Fine-tuning → Model adapters & registry: model family behavior
- MT-Bench answer generation → Controller: queries local models
- MT-Bench answer generation → Remote provider adapter: queries APIs
- MT-Bench answer generation → MT-Bench judging & results: generated answers
- Multi-model Arena UI → Arena analytics: conversation and vote logs
- Docker deployment → Controller: deploys
- Docker deployment → Transformers worker: deploys
