How antgroup/cakekv works
Project type and purpose
• This repository implements “CAKE”: a research‐driven, open‑source Python library and set of evaluation scripts for Cascading and Adaptive KV Cache Eviction in large language model (LLM) inference.
• Its primary goal is to provide a modular cache‐eviction manager that plugs into Hugging Face–style models (e.g., LLaMA, Mistral, Qwen2), apply layer‑aware cache allocation during generation, and evaluate on benchmarks like LongBench.
File structure
• Top‑level:
• cake/ ← core Python package
• experiments/ ← evaluation and benchmarking scripts
• images/ ← illustrations referenced in README
• install.sh, pyproject.toml, README.md, etc.
• cake/ package:
• cake_cache.py ← main cache manager and eviction logic
• model/ ← per‑model monkey‑patchers (modify_llama.py, modify_mistral.py, modify_qwen2.py)
• monkeypatch.py ← integration entrypoint that applies patches to HF models
• utils.py ← supporting functions (e.g., metrics, common helpers)
• __init__.py
• experiments/LongBench:
• config/ ← JSON files mapping datasets to prompt length, models to cache sizes, etc.
• pred_cake.py ← runs generation under CAKE’s eviction
• eval.py ← computes metrics on predictions
• metrics.py
• Configuration files (dataset2*.json, model2*.json) define hyperparameters externally.
README for additional insights
• Describes algorithm: cascading layers, adaptive eviction indicator, rational cache allocation.
• Quick Start uses Hugging Face transformers package; key dependency pinned at transformers==4.43.3.
• Illustrations show attention‑pattern analysis and cache allocation process.
a. Main components
1. CLI / Experiment Runner
– pred_cake.py and eval.py parse args, load configs, launch runs.
2. Configuration subsystem
– JSON files (dataset2maxlen, model2maxlen, model2path, model2tau)
3. CAKE core library
– CacheManager (cake_cache.py): eviction indicator, cascading allocator
– ModelAdapter (monkeypatch.py + modify_*.py): hooks into HF model’s forward/prefill
– Utils (utils.py): common support
4. External frameworks
– Hugging Face Transformers (model loading, tokenization)
– PyTorch (tensor ops, memory management)
5. Output & Metrics
– Predicted outputs (saved by pred_cake.py)
– Evaluation results (eval.py → metrics.py)
b. Relationships and interactions
• Experiment runner reads config → loads model via HF API
• ModelAdapter monkey‑patches caching hooks into model’s generate()/prefill
• CacheManager invoked at each layer’s attention call to decide eviction
• Token streams flow through patched model → eviction logic consults layer preferences
• Predictions written to disk → eval module ingests them and computes metrics
c. Architectural patterns and design principles
• Plugin/monkey‑patch architecture for model‑agnostic integration
• Separation of concerns: core eviction logic separated from model specifics
• Config‑driven experimentation to avoid hard‑coding hyperparameters
d. Key technologies and libraries
• Python 3.x, PyTorch, Hugging Face Transformers
• JSON for experiment config
• Shell script for environment setup (install.sh)
Detail
• Which JSON config maps to which internal parameter
• That monkeypatch.py is the single integration entrypoint
• Call out performance metrics flow (predictions → metrics.py)