How 1catai/1cat-vllm works
1Cat-vLLM is a vLLM fork focused on running modern LLM inference efficiently on NVIDIA Volta/SM70, while retaining broader model-serving capabilities. The principal workflow is a client request entering the serving layer, being validated and rendered into engine input, executed by the inference engine and model kernels, then returned as generated output. The README highlights optimized long-context attention, quantized models/KV cache, and speculative decoding as major performance branches. The tree and excerpts establish the main components, but do not expose all engine-core and speculative-decoding wiring; uncertain links are omitted.
Main entry points: model_executor Model Execution · engine Engine Core · routes HTTP Routes · client.rs Engine Client
Components 18
Serving API
- HTTP Routes
rust/src/server/src/routes - Engine Client
rust/src/engine-core-client/src/client.rs
Request Processing
- Chat Handling
rust/src/chat/src - Prompt Renderer
rust/src/chat/src/renderer - Tokenizers
rust/src/tokenizer/src - Generated Output
rust/src/llm/src/output.rs
Inference Runtime
- LLM Engine
vllm/v1/engine/llm_engine.py - Engine Inputs
vllm/inputs/engine.py - Engine Core
vllm/v1/engine - Model Execution
vllm/model_executor - Speculative Decode
vllm/v1
Model Kernels
- SM70 Attention
csrc/attention - Flash-V100
flash-attention-v100/kernel - Quantized Kernels
csrc/quantization - MoE Kernels
csrc/moe
Parallel State
- KV Cache
csrc/cache.h - KV Transfer
vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py
Other components
- API Client
Connections 18
- API Client → HTTP Routes: sends request
- HTTP Routes → Chat Handling: dispatches chat
- Chat Handling → Prompt Renderer: renders prompt
- Prompt Renderer → Tokenizers: tokenizes prompt
- HTTP Routes → Engine Client: submits request
- Engine Client → LLM Engine: forwards request
- LLM Engine → Engine Inputs: constructs input
- LLM Engine → Engine Core: dispatches work
- Engine Core → Model Execution: executes inference
- Model Execution → SM70 Attention: computes attention
- Model Execution → Quantized Kernels: runs quantization
- Model Execution → MoE Kernels: runs experts
- SM70 Attention → Flash-V100: uses kernels
- Model Execution → KV Cache: reads/writes cache
- Engine Core → Speculative Decode: may dispatch
- Engine Core → KV Transfer: may transfer KV
- Engine Client → Generated Output: returns outputs
- Generated Output → API Client: delivers result