How yangnobody12/fast-serve works
fast-serve is a Rust/CUDA inference server for Qwen3.5-4B. A client submits chat messages to the OpenAI-compatible API; the server formats and tokenizes prompts, runs either the single-user split engine or continuous-batching engine, and streams generated tokens as SSE. Both engines divide model execution across two GPU stages, backed by packed BF16 weights and CUDA kernels; the batched mode adds paged KV allocation. The README documents these modules and capabilities, but unsampled source limits confidence about some internal call relationships.
Main entry points: batch_engine.rs Continuous Batching · engine.rs Split Inference · api.rs HTTP and SSE API · cuda_ffi.rs CUDA FFI Runtime
Components 12
Serving API
- HTTP and SSE API
src/api.rs - Server Entry
src/main.rs
Inference Engines
- Split Inference
src/engine.rs - Continuous Batching
src/batch_engine.rs - Paged KV Pool
src/kv_pool.rs
GPU Runtime
- CUDA FFI Runtime
src/cuda_ffi.rs - CUDA Kernels
kernels/qwen35.cu
Model Inputs
- Qwen Tokenizer
src/tokenizer.rs - BF16 Weight Loader
src/loader.rs - Model Config
src/config.rs
Other components
- Chat Client
- NVIDIA GPU
Connections 21
- Chat Client → HTTP and SSE API: submits chat
README.md - HTTP and SSE API → Chat Client: streams SSE
README.md - Server Entry → HTTP and SSE API: builds router
src/main.rs - Server Entry → Continuous Batching: loads batched mode
src/main.rs - Server Entry → Split Inference: loads single mode
README.md - HTTP and SSE API → Split Inference: serves single mode
README.md - HTTP and SSE API → Continuous Batching: submits batched requests
src/api.rs - Split Inference → BF16 Weight Loader: loads weights
README.md - Continuous Batching → BF16 Weight Loader: loads weights
README.md - BF16 Weight Loader → Model Config: reads model settings
src/loader.rs - Split Inference → Model Config: uses model settings
README.md - Continuous Batching → Model Config: uses model settings
README.md - Split Inference → Qwen Tokenizer: tokenizes prompts
README.md - Continuous Batching → Qwen Tokenizer: tokenizes prompts
README.md - Continuous Batching → Paged KV Pool: allocates paged KV
README.md - Split Inference → CUDA FFI Runtime: launches GPU work
src/engine.rs - Continuous Batching → CUDA FFI Runtime: launches GPU work
src/batch_engine.rs - BF16 Weight Loader → CUDA FFI Runtime: uploads weights
src/loader.rs - CUDA FFI Runtime → CUDA Kernels: dispatches kernels
README.md - CUDA FFI Runtime → NVIDIA GPU: uses CUDA driver
src/cuda_ffi.rs - CUDA Kernels → NVIDIA GPU: executes on GPU
README.md