How ydyhello/vecinfer works
VecInfer is an LLM inference system that compresses key/value caches with outlier-suppressed product quantization, alongside codebook preparation and accuracy/efficiency evaluation. The documented workflow calibrates key-channel scales, samples cache vectors, trains or selects codebooks, then runs a configured model with a compressed cache and optimized attention kernels. The graph combines README capability descriptions with the sampled entry point and model/cache code; evaluation and calibration wiring is shown only at the subsystem level because those implementations were not sampled.
Main entry points: main_pq.py Pipeline Runner · ModelContext.py Model Context · modeling_mistral.py Model Adapters · evaluation Benchmark Evaluation
Components 18
Calibration and Codebooks
- Scale Calibration
scripts/utils/calibration.py - KV Samples
- Codebooks
- Vector I/O
scripts/utils/fvecio.py
Model Inference
- Pipeline Runner
scripts/modeldb/main_pq.py - Model Context
scripts/modeldb/models/ModelContext.py - Model Adapters
scripts/modeldb/models/modeling_mistral.py - Llama Adapter
scripts/modeldb/models/modeling_llama.py - Qwen2 Adapter
scripts/modeldb/models/modeling_qwen2.py - Qwen3 Adapter
scripts/modeldb/models/modeling_qwen3.py - PQ Cache
scripts/utils/pq_utils.py
Quantized Attention
- CUDA Extension
scripts/modeldb/bindings/bindings.cpp
Evaluation
- Benchmark Evaluation
scripts/utils/evaluation - Math Answer Parsing
scripts/utils/evaluation/parser.py - Evaluation Results
Other components
- Researcher
- Model Weights
- Datasets
Connections 21
- Researcher → Scale Calibration: calibrates scales
- Model Weights → Scale Calibration: provides model
- Scale Calibration → Pipeline Runner: supplies scales
- Researcher → Pipeline Runner: configures and runs
- Datasets → Pipeline Runner: provides inputs
- Pipeline Runner → Model Context: loads context
- Model Context → Model Adapters: resolves model
- Model Context → Llama Adapter: resolves model
- Model Context → Qwen2 Adapter: resolves model
- Model Context → Qwen3 Adapter: resolves model
- Model Weights → Model Adapters: loads weights
- Model Adapters → PQ Cache: updates cache
- Pipeline Runner → KV Samples: writes sampled vectors
- Pipeline Runner → Vector I/O: reads vectors
- Vector I/O → KV Samples: reads and writes
- Pipeline Runner → Codebooks: writes trained codebooks
- Pipeline Runner → Benchmark Evaluation: runs benchmarks
- Benchmark Evaluation → Math Answer Parsing: normalizes answers
- Benchmark Evaluation → Evaluation Results: saves scores
- Model Adapters → CUDA Extension: fused attention
- Researcher → Benchmark Evaluation: selects benchmark