# 1catai/1cat-vllm architecture

> Architecture diagram of the GitHub repository 1catai/1cat-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/1catai/1cat-vllm
- Repository: https://github.com/1catai/1cat-vllm

## Overview

1Cat-vLLM is a vLLM fork focused on running modern LLM inference efficiently on NVIDIA Volta/SM70, while retaining broader model-serving capabilities. The principal workflow is a client request entering the serving layer, being validated and rendered into engine input, executed by the inference engine and model kernels, then returned as generated output. The README highlights optimized long-context attention, quantized models/KV cache, and speculative decoding as major performance branches. The tree and excerpts establish the main components, but do not expose all engine-core and speculative-decoding wiring; uncertain links are omitted.

Main entry points: [`vllm/model_executor`](https://github.com/1catai/1cat-vllm/tree/main/vllm/model_executor) (Model Execution), [`vllm/v1/engine`](https://github.com/1catai/1cat-vllm/tree/main/vllm/v1/engine) (Engine Core), [`rust/src/server/src/routes`](https://github.com/1catai/1cat-vllm/tree/main/rust/src/server/src/routes) (HTTP Routes), [`rust/src/engine-core-client/src/client.rs`](https://github.com/1catai/1cat-vllm/blob/main/rust/src/engine-core-client/src/client.rs) (Engine Client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving API"]
  node_api["HTTP Routes"]
  node_engine_client["Engine Client<br/>[client.rs]"]
end

subgraph group_request["Request Processing"]
  node_chat["Chat Handling"]
  node_renderer["Prompt Renderer"]
  node_tokenizer["Tokenizers"]
  node_output["Generated Output<br/>[output.rs]"]
end

subgraph group_runtime["Inference Runtime"]
  node_py_engine["LLM Engine<br/>[llm_engine.py]"]
  node_input_schema["Engine Inputs<br/>[engine.py]"]
  node_engine_core["Engine Core"]
  node_model["Model Execution"]
  node_spec_decode["Speculative Decode"]
end

subgraph group_kernels["Model Kernels"]
  node_attention["SM70 Attention"]
  node_v100_flash["Flash-V100"]
  node_quant["Quantized Kernels"]
  node_moe["MoE Kernels"]
end

subgraph group_parallel["Parallel State"]
  node_kv[("KV Cache<br/>[cache.h]")]
  node_kv_transfer["KV Transfer<br/>[moriio_engine.py]"]
end

node_client(("API Client"))

node_client -->|"sends request"| node_api
node_api -->|"dispatches chat"| node_chat
node_chat -->|"renders prompt"| node_renderer
node_renderer -->|"tokenizes prompt"| node_tokenizer
node_api -->|"submits request"| node_engine_client
node_engine_client -->|"forwards request"| node_py_engine
node_py_engine -->|"constructs input"| node_input_schema
node_py_engine -->|"dispatches work"| node_engine_core
node_engine_core -->|"executes inference"| node_model
node_model -->|"computes attention"| node_attention
node_model -->|"runs quantization"| node_quant
node_model -->|"runs experts"| node_moe
node_attention -.->|"uses kernels"| node_v100_flash
node_model -->|"reads/writes cache"| node_kv
node_engine_core -.->|"may dispatch"| node_spec_decode
node_engine_core -.->|"may transfer KV"| node_kv_transfer
node_engine_client -->|"returns outputs"| node_output
node_output -->|"delivers result"| node_client

click node_api "https://github.com/1catai/1cat-vllm/tree/main/rust/src/server/src/routes"
click node_chat "https://github.com/1catai/1cat-vllm/tree/main/rust/src/chat/src"
click node_renderer "https://github.com/1catai/1cat-vllm/tree/main/rust/src/chat/src/renderer"
click node_tokenizer "https://github.com/1catai/1cat-vllm/tree/main/rust/src/tokenizer/src"
click node_engine_client "https://github.com/1catai/1cat-vllm/blob/main/rust/src/engine-core-client/src/client.rs"
click node_py_engine "https://github.com/1catai/1cat-vllm/blob/main/vllm/v1/engine/llm_engine.py"
click node_input_schema "https://github.com/1catai/1cat-vllm/blob/main/vllm/inputs/engine.py"
click node_engine_core "https://github.com/1catai/1cat-vllm/tree/main/vllm/v1/engine"
click node_model "https://github.com/1catai/1cat-vllm/tree/main/vllm/model_executor"
click node_attention "https://github.com/1catai/1cat-vllm/tree/main/csrc/attention"
click node_v100_flash "https://github.com/1catai/1cat-vllm/tree/main/flash-attention-v100/kernel"
click node_quant "https://github.com/1catai/1cat-vllm/tree/main/csrc/quantization"
click node_moe "https://github.com/1catai/1cat-vllm/tree/main/csrc/moe"
click node_kv "https://github.com/1catai/1cat-vllm/blob/main/csrc/cache.h"
click node_kv_transfer "https://github.com/1catai/1cat-vllm/blob/main/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py"
click node_spec_decode "https://github.com/1catai/1cat-vllm/tree/main/vllm/v1"
click node_output "https://github.com/1catai/1cat-vllm/blob/main/rust/src/llm/src/output.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_engine_client,node_client toneBlue
class node_chat,node_renderer,node_tokenizer,node_output toneAmber
class node_py_engine,node_input_schema,node_engine_core,node_model,node_spec_decode toneMint
class node_attention,node_v100_flash,node_quant,node_moe toneRose
class node_kv,node_kv_transfer toneIndigo
```

## Components

### Serving API

- [HTTP Routes](https://github.com/1catai/1cat-vllm/tree/main/rust/src/server/src/routes): `rust/src/server/src/routes`
- [Engine Client](https://github.com/1catai/1cat-vllm/blob/main/rust/src/engine-core-client/src/client.rs): `rust/src/engine-core-client/src/client.rs`

### Request Processing

- [Chat Handling](https://github.com/1catai/1cat-vllm/tree/main/rust/src/chat/src): `rust/src/chat/src`
- [Prompt Renderer](https://github.com/1catai/1cat-vllm/tree/main/rust/src/chat/src/renderer): `rust/src/chat/src/renderer`
- [Tokenizers](https://github.com/1catai/1cat-vllm/tree/main/rust/src/tokenizer/src): `rust/src/tokenizer/src`
- [Generated Output](https://github.com/1catai/1cat-vllm/blob/main/rust/src/llm/src/output.rs): `rust/src/llm/src/output.rs`

### Inference Runtime

- [LLM Engine](https://github.com/1catai/1cat-vllm/blob/main/vllm/v1/engine/llm_engine.py): `vllm/v1/engine/llm_engine.py`
- [Engine Inputs](https://github.com/1catai/1cat-vllm/blob/main/vllm/inputs/engine.py): `vllm/inputs/engine.py`
- [Engine Core](https://github.com/1catai/1cat-vllm/tree/main/vllm/v1/engine): `vllm/v1/engine`
- [Model Execution](https://github.com/1catai/1cat-vllm/tree/main/vllm/model_executor): `vllm/model_executor`
- [Speculative Decode](https://github.com/1catai/1cat-vllm/tree/main/vllm/v1): `vllm/v1`

### Model Kernels

- [SM70 Attention](https://github.com/1catai/1cat-vllm/tree/main/csrc/attention): `csrc/attention`
- [Flash-V100](https://github.com/1catai/1cat-vllm/tree/main/flash-attention-v100/kernel): `flash-attention-v100/kernel`
- [Quantized Kernels](https://github.com/1catai/1cat-vllm/tree/main/csrc/quantization): `csrc/quantization`
- [MoE Kernels](https://github.com/1catai/1cat-vllm/tree/main/csrc/moe): `csrc/moe`

### Parallel State

- [KV Cache](https://github.com/1catai/1cat-vllm/blob/main/csrc/cache.h): `csrc/cache.h`
- [KV Transfer](https://github.com/1catai/1cat-vllm/blob/main/vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py): `vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py`

### Other components

- **API Client**

## Connections

- API Client → HTTP Routes: sends request
- HTTP Routes → Chat Handling: dispatches chat
- Chat Handling → Prompt Renderer: renders prompt
- Prompt Renderer → Tokenizers: tokenizes prompt
- HTTP Routes → Engine Client: submits request
- Engine Client → LLM Engine: forwards request
- LLM Engine → Engine Inputs: constructs input
- LLM Engine → Engine Core: dispatches work
- Engine Core → Model Execution: executes inference
- Model Execution → SM70 Attention: computes attention
- Model Execution → Quantized Kernels: runs quantization
- Model Execution → MoE Kernels: runs experts
- SM70 Attention → Flash-V100: uses kernels
- Model Execution → KV Cache: reads/writes cache
- Engine Core → Speculative Decode: may dispatch
- Engine Core → KV Transfer: may transfer KV
- Engine Client → Generated Output: returns outputs
- Generated Output → API Client: delivers result