# yangnobody12/fast-serve architecture

> Architecture diagram of the GitHub repository yangnobody12/fast-serve, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yangnobody12/fast-serve
- Repository: https://github.com/yangnobody12/fast-serve

## Overview

fast-serve is a Rust/CUDA inference server for Qwen3.5-4B. A client submits chat messages to the OpenAI-compatible API; the server formats and tokenizes prompts, runs either the single-user split engine or continuous-batching engine, and streams generated tokens as SSE. Both engines divide model execution across two GPU stages, backed by packed BF16 weights and CUDA kernels; the batched mode adds paged KV allocation. The README documents these modules and capabilities, but unsampled source limits confidence about some internal call relationships.

Main entry points: [`src/batch_engine.rs`](https://github.com/yangnobody12/fast-serve/blob/main/src/batch_engine.rs) (Continuous Batching), [`src/engine.rs`](https://github.com/yangnobody12/fast-serve/blob/main/src/engine.rs) (Split Inference), [`src/api.rs`](https://github.com/yangnobody12/fast-serve/blob/main/src/api.rs) (HTTP and SSE API), [`src/cuda_ffi.rs`](https://github.com/yangnobody12/fast-serve/blob/main/src/cuda_ffi.rs) (CUDA FFI Runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving API"]
  node_api["HTTP and SSE API<br/>[api.rs]"]
  node_entry["Server Entry<br/>[main.rs]"]
end

subgraph group_inference["Inference Engines"]
  node_single["Split Inference<br/>[engine.rs]"]
  node_batch["Continuous Batching<br/>[batch_engine.rs]"]
  node_kv[("Paged KV Pool<br/>[kv_pool.rs]")]
end

subgraph group_gpu["GPU Runtime"]
  node_cuda["CUDA FFI Runtime<br/>[cuda_ffi.rs]"]
  node_kernels["CUDA Kernels<br/>[qwen35.cu]"]
end

subgraph group_model["Model Inputs"]
  node_tokenizer["Qwen Tokenizer<br/>[tokenizer.rs]"]
  node_loader["BF16 Weight Loader<br/>[loader.rs]"]
  node_config["Model Config<br/>[config.rs]"]
end

node_client(("Chat Client"))
node_gpu_device["NVIDIA GPU"]

node_client -->|"submits chat"| node_api
node_api -->|"streams SSE"| node_client
node_entry -->|"builds router"| node_api
node_entry -->|"loads batched mode"| node_batch
node_entry -->|"loads single mode"| node_single
node_api -->|"serves single mode"| node_single
node_api -->|"submits batched requests"| node_batch
node_single -->|"loads weights"| node_loader
node_batch -->|"loads weights"| node_loader
node_loader -->|"reads model settings"| node_config
node_single -->|"uses model settings"| node_config
node_batch -->|"uses model settings"| node_config
node_single -->|"tokenizes prompts"| node_tokenizer
node_batch -->|"tokenizes prompts"| node_tokenizer
node_batch -->|"allocates paged KV"| node_kv
node_single -->|"launches GPU work"| node_cuda
node_batch -->|"launches GPU work"| node_cuda
node_loader -->|"uploads weights"| node_cuda
node_cuda -->|"dispatches kernels"| node_kernels
node_cuda -->|"uses CUDA driver"| node_gpu_device
node_kernels -->|"executes on GPU"| node_gpu_device

click node_api "https://github.com/yangnobody12/fast-serve/blob/main/src/api.rs"
click node_entry "https://github.com/yangnobody12/fast-serve/blob/main/src/main.rs"
click node_tokenizer "https://github.com/yangnobody12/fast-serve/blob/main/src/tokenizer.rs"
click node_single "https://github.com/yangnobody12/fast-serve/blob/main/src/engine.rs"
click node_batch "https://github.com/yangnobody12/fast-serve/blob/main/src/batch_engine.rs"
click node_kv "https://github.com/yangnobody12/fast-serve/blob/main/src/kv_pool.rs"
click node_loader "https://github.com/yangnobody12/fast-serve/blob/main/src/loader.rs"
click node_config "https://github.com/yangnobody12/fast-serve/blob/main/src/config.rs"
click node_cuda "https://github.com/yangnobody12/fast-serve/blob/main/src/cuda_ffi.rs"
click node_kernels "https://github.com/yangnobody12/fast-serve/blob/main/kernels/qwen35.cu"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_entry,node_client toneBlue
class node_single,node_batch,node_kv toneAmber
class node_cuda,node_kernels toneMint
class node_tokenizer,node_loader,node_config toneRose
class node_gpu_device toneIndigo
```

## Components

### Serving API

- [HTTP and SSE API](https://github.com/yangnobody12/fast-serve/blob/main/src/api.rs): `src/api.rs`
- [Server Entry](https://github.com/yangnobody12/fast-serve/blob/main/src/main.rs): `src/main.rs`

### Inference Engines

- [Split Inference](https://github.com/yangnobody12/fast-serve/blob/main/src/engine.rs): `src/engine.rs`
- [Continuous Batching](https://github.com/yangnobody12/fast-serve/blob/main/src/batch_engine.rs): `src/batch_engine.rs`
- [Paged KV Pool](https://github.com/yangnobody12/fast-serve/blob/main/src/kv_pool.rs): `src/kv_pool.rs`

### GPU Runtime

- [CUDA FFI Runtime](https://github.com/yangnobody12/fast-serve/blob/main/src/cuda_ffi.rs): `src/cuda_ffi.rs`
- [CUDA Kernels](https://github.com/yangnobody12/fast-serve/blob/main/kernels/qwen35.cu): `kernels/qwen35.cu`

### Model Inputs

- [Qwen Tokenizer](https://github.com/yangnobody12/fast-serve/blob/main/src/tokenizer.rs): `src/tokenizer.rs`
- [BF16 Weight Loader](https://github.com/yangnobody12/fast-serve/blob/main/src/loader.rs): `src/loader.rs`
- [Model Config](https://github.com/yangnobody12/fast-serve/blob/main/src/config.rs): `src/config.rs`

### Other components

- **Chat Client**
- **NVIDIA GPU**

## Connections

- Chat Client → HTTP and SSE API: submits chat (evidence: `README.md`)
- HTTP and SSE API → Chat Client: streams SSE (evidence: `README.md`)
- Server Entry → HTTP and SSE API: builds router (evidence: `src/main.rs`)
- Server Entry → Continuous Batching: loads batched mode (evidence: `src/main.rs`)
- Server Entry → Split Inference: loads single mode (evidence: `README.md`)
- HTTP and SSE API → Split Inference: serves single mode (evidence: `README.md`)
- HTTP and SSE API → Continuous Batching: submits batched requests (evidence: `src/api.rs`)
- Split Inference → BF16 Weight Loader: loads weights (evidence: `README.md`)
- Continuous Batching → BF16 Weight Loader: loads weights (evidence: `README.md`)
- BF16 Weight Loader → Model Config: reads model settings (evidence: `src/loader.rs`)
- Split Inference → Model Config: uses model settings (evidence: `README.md`)
- Continuous Batching → Model Config: uses model settings (evidence: `README.md`)
- Split Inference → Qwen Tokenizer: tokenizes prompts (evidence: `README.md`)
- Continuous Batching → Qwen Tokenizer: tokenizes prompts (evidence: `README.md`)
- Continuous Batching → Paged KV Pool: allocates paged KV (evidence: `README.md`)
- Split Inference → CUDA FFI Runtime: launches GPU work (evidence: `src/engine.rs`)
- Continuous Batching → CUDA FFI Runtime: launches GPU work (evidence: `src/batch_engine.rs`)
- BF16 Weight Loader → CUDA FFI Runtime: uploads weights (evidence: `src/loader.rs`)
- CUDA FFI Runtime → CUDA Kernels: dispatches kernels (evidence: `README.md`)
- CUDA FFI Runtime → NVIDIA GPU: uses CUDA driver (evidence: `src/cuda_ffi.rs`)
- CUDA Kernels → NVIDIA GPU: executes on GPU (evidence: `README.md`)
