# ydyhello/vecinfer architecture

> Architecture diagram of the GitHub repository ydyhello/vecinfer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ydyhello/vecinfer
- Repository: https://github.com/ydyhello/vecinfer

## Overview

VecInfer is an LLM inference system that compresses key/value caches with outlier-suppressed product quantization, alongside codebook preparation and accuracy/efficiency evaluation. The documented workflow calibrates key-channel scales, samples cache vectors, trains or selects codebooks, then runs a configured model with a compressed cache and optimized attention kernels. The graph combines README capability descriptions with the sampled entry point and model/cache code; evaluation and calibration wiring is shown only at the subsystem level because those implementations were not sampled.

Main entry points: [`scripts/modeldb/main_pq.py`](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/main_pq.py) (Pipeline Runner), [`scripts/modeldb/models/ModelContext.py`](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/ModelContext.py) (Model Context), [`scripts/modeldb/models/modeling_mistral.py`](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_mistral.py) (Model Adapters), [`scripts/utils/evaluation`](https://github.com/ydyhello/vecinfer/tree/main/scripts/utils/evaluation) (Benchmark Evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Calibration and Codebooks"]
  node_calibration["Scale Calibration<br/>[calibration.py]"]
  node_samples["KV Samples"]
  node_codebooks["Codebooks"]
  node_fvecio["Vector I/O<br/>[fvecio.py]"]
end

subgraph group_inference["Model Inference"]
  node_runner["Pipeline Runner<br/>[main_pq.py]"]
  node_contexts["Model Context<br/>[ModelContext.py]"]
  node_familymodels["Model Adapters"]
  node_llama["Llama Adapter<br/>[modeling_llama.py]"]
  node_qwen2["Qwen2 Adapter<br/>[modeling_qwen2.py]"]
  node_qwen3["Qwen3 Adapter<br/>[modeling_qwen3.py]"]
  node_pqcache["PQ Cache<br/>[pq_utils.py]"]
end

subgraph group_kernel["Quantized Attention"]
  node_native["CUDA Extension<br/>[bindings.cpp]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval["Benchmark Evaluation"]
  node_mathparser["Math Answer Parsing<br/>[parser.py]"]
  node_results["Evaluation Results"]
end

node_user(("Researcher"))
node_modeldata["Model Weights"]
node_datasets["Datasets"]

node_user -->|"calibrates scales"| node_calibration
node_modeldata -->|"provides model"| node_calibration
node_calibration -->|"supplies scales"| node_runner
node_user -->|"configures and runs"| node_runner
node_datasets -->|"provides inputs"| node_runner
node_runner -->|"loads context"| node_contexts
node_contexts -->|"resolves model"| node_familymodels
node_contexts -->|"resolves model"| node_llama
node_contexts -->|"resolves model"| node_qwen2
node_contexts -->|"resolves model"| node_qwen3
node_modeldata -->|"loads weights"| node_familymodels
node_familymodels -->|"updates cache"| node_pqcache
node_runner -->|"writes sampled vectors"| node_samples
node_runner -->|"reads vectors"| node_fvecio
node_fvecio -->|"reads and writes"| node_samples
node_runner -->|"writes trained codebooks"| node_codebooks
node_runner -->|"runs benchmarks"| node_eval
node_eval -->|"normalizes answers"| node_mathparser
node_eval -->|"saves scores"| node_results
node_familymodels -.->|"fused attention"| node_native
node_user -->|"selects benchmark"| node_eval

click node_calibration "https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/calibration.py"
click node_runner "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/main_pq.py"
click node_contexts "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/ModelContext.py"
click node_familymodels "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_mistral.py"
click node_llama "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_llama.py"
click node_qwen2 "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_qwen2.py"
click node_qwen3 "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_qwen3.py"
click node_pqcache "https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/pq_utils.py"
click node_native "https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/bindings/bindings.cpp"
click node_fvecio "https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/fvecio.py"
click node_eval "https://github.com/ydyhello/vecinfer/tree/main/scripts/utils/evaluation"
click node_mathparser "https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/evaluation/parser.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_calibration,node_samples,node_codebooks,node_fvecio toneBlue
class node_runner,node_contexts,node_familymodels,node_llama,node_qwen2,node_qwen3,node_pqcache toneAmber
class node_native toneMint
class node_eval,node_mathparser,node_results toneRose
class node_user,node_modeldata,node_datasets toneIndigo
```

## Components

### Calibration and Codebooks

- [Scale Calibration](https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/calibration.py): `scripts/utils/calibration.py`
- **KV Samples**
- **Codebooks**
- [Vector I/O](https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/fvecio.py): `scripts/utils/fvecio.py`

### Model Inference

- [Pipeline Runner](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/main_pq.py): `scripts/modeldb/main_pq.py`
- [Model Context](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/ModelContext.py): `scripts/modeldb/models/ModelContext.py`
- [Model Adapters](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_mistral.py): `scripts/modeldb/models/modeling_mistral.py`
- [Llama Adapter](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_llama.py): `scripts/modeldb/models/modeling_llama.py`
- [Qwen2 Adapter](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_qwen2.py): `scripts/modeldb/models/modeling_qwen2.py`
- [Qwen3 Adapter](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/models/modeling_qwen3.py): `scripts/modeldb/models/modeling_qwen3.py`
- [PQ Cache](https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/pq_utils.py): `scripts/utils/pq_utils.py`

### Quantized Attention

- [CUDA Extension](https://github.com/ydyhello/vecinfer/blob/main/scripts/modeldb/bindings/bindings.cpp): `scripts/modeldb/bindings/bindings.cpp`

### Evaluation

- [Benchmark Evaluation](https://github.com/ydyhello/vecinfer/tree/main/scripts/utils/evaluation): `scripts/utils/evaluation`
- [Math Answer Parsing](https://github.com/ydyhello/vecinfer/blob/main/scripts/utils/evaluation/parser.py): `scripts/utils/evaluation/parser.py`
- **Evaluation Results**

### Other components

- **Researcher**
- **Model Weights**
- **Datasets**

## Connections

- Researcher → Scale Calibration: calibrates scales
- Model Weights → Scale Calibration: provides model
- Scale Calibration → Pipeline Runner: supplies scales
- Researcher → Pipeline Runner: configures and runs
- Datasets → Pipeline Runner: provides inputs
- Pipeline Runner → Model Context: loads context
- Model Context → Model Adapters: resolves model
- Model Context → Llama Adapter: resolves model
- Model Context → Qwen2 Adapter: resolves model
- Model Context → Qwen3 Adapter: resolves model
- Model Weights → Model Adapters: loads weights
- Model Adapters → PQ Cache: updates cache
- Pipeline Runner → KV Samples: writes sampled vectors
- Pipeline Runner → Vector I/O: reads vectors
- Vector I/O → KV Samples: reads and writes
- Pipeline Runner → Codebooks: writes trained codebooks
- Pipeline Runner → Benchmark Evaluation: runs benchmarks
- Benchmark Evaluation → Math Answer Parsing: normalizes answers
- Benchmark Evaluation → Evaluation Results: saves scores
- Model Adapters → CUDA Extension: fused attention
- Researcher → Benchmark Evaluation: selects benchmark