# antgroup/cakekv architecture

> Architecture diagram of the GitHub repository antgroup/cakekv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antgroup/cakekv
- Repository: https://github.com/antgroup/cakekv

## Overview

##### Project type and purpose
- This repository implements “CAKE”: a research‐driven, open‑source Python library and set of evaluation scripts for Cascading and Adaptive KV Cache Eviction in large language model (LLM) inference.
- Its primary goal is to provide a modular cache‐eviction manager that plugs into Hugging Face–style models (e.g., LLaMA, Mistral, Qwen2), apply layer‑aware cache allocation during generation, and evaluate on benchmarks like LongBench.

##### File structure
- Top‑level:
- cake/           ← core Python package
- experiments/    ← evaluation and benchmarking scripts
- images/         ← illustrations referenced in README
- install.sh, pyproject.toml, README.md, etc.
- cake/ package:
- cake_cache.py   ← main cache manager and eviction logic
- model/          ← per‑model monkey‑patchers (modify_llama.py, modify_mistral.py, modify_qwen2.py)
- monkeypatch.py  ← integration entrypoint that applies patches to HF models
- utils.py        ← supporting functions (e.g., metrics, common helpers)
- __init__.py
- experiments/LongBench:
- config/         ← JSON files mapping datasets to prompt length, models to cache sizes, etc.
- pred_cake.py    ← runs generation under CAKE’s eviction
- eval.py         ← computes metrics on predictions
- metrics.py
- Configuration files (dataset2*.json, model2*.json) define hyperparameters externally.

##### README for additional insights
- Describes algorithm: cascading layers, adaptive eviction indicator, rational cache allocation.
- Quick Start uses Hugging Face transformers package; key dependency pinned at transformers==4.43.3.
- Illustrations show attention‑pattern analysis and cache allocation process.

a. Main components
1. CLI / Experiment Runner
– pred_cake.py and eval.py parse args, load configs, launch runs.
2. Configuration subsystem
– JSON files (dataset2maxlen, model2maxlen, model2path, model2tau)
3. CAKE core library
– CacheManager (cake_cache.py): eviction indicator, cascading allocator
– ModelAdapter (monkeypatch.py + modify_*.py): hooks into HF model’s forward/prefill
– Utils (utils.py): common support
4. External frameworks
– Hugging Face Transformers (model loading, tokenization)
– PyTorch (tensor ops, memory management)
5. Output & Metrics
– Predicted outputs (saved by pred_cake.py)
– Evaluation results (eval.py → metrics.py)
b. Relationships and interactions
- Experiment runner reads config → loads model via HF API
- ModelAdapter monkey‑patches caching hooks into model’s generate()/prefill
- CacheManager invoked at each layer’s attention call to decide eviction
- Token streams flow through patched model → eviction logic consults layer preferences
- Predictions written to disk → eval module ingests them and computes metrics
c. Architectural patterns and design principles
- Plugin/monkey‑patch architecture for model‑agnostic integration
- Separation of concerns: core eviction logic separated from model specifics
- Config‑driven experimentation to avoid hard‑coding hyperparameters
d. Key technologies and libraries
- Python 3.x, PyTorch, Hugging Face Transformers
- JSON for experiment config
- Shell script for environment setup (install.sh)

##### Detail
- Which JSON config maps to which internal parameter
- That monkeypatch.py is the single integration entrypoint
- Call out performance metrics flow (predictions → metrics.py)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Experiment Layer
    subgraph "Experiment Layer"
        direction TB
        PredCake["pred_cake.py"]:::experiment
        Eval["eval.py"]:::experiment
        MetricsScript["metrics.py"]:::experiment
    end

    %% Configuration Subsystem
    subgraph "Configuration Subsystem"
        direction TB
        DSMaxLen["dataset2maxlen.json"]:::config
        DSPrompt["dataset2prompt.json"]:::config
        ModelMaxLen["model2maxlen.json"]:::config
        ModelPathMap["model2path.json"]:::config
        ModelTau["model2tau.json"]:::config
    end

    %% Core Library
    subgraph "CAKE Core Library"
        direction TB
        CacheManager["CacheManager\n(cake_cache.py)"]:::core
        ModelAdapter["ModelAdapter\n(monkeypatch.py)"]:::core
        Utils["utils.py"]:::core
        subgraph "Per-Model Adapters"
            direction TB
            LLamaAdapter["modify_llama.py"]:::core
            MistralAdapter["modify_mistral.py"]:::core
            Qwen2Adapter["modify_qwen2.py"]:::core
        end
    end

    %% External Frameworks
    subgraph "External Frameworks"
        direction TB
        HF["Hugging Face\nTransformers"]:::external
        Torch["PyTorch"]:::external
    end

    %% Output & Metrics
    subgraph "Output & Metrics"
        direction TB
        Predictions["Prediction Outputs"]:::output
        EvalResults["Evaluation Results"]:::output
    end

    %% Connections
    PredCake -->|"reads"| DSMaxLen
    PredCake -->|"reads"| DSPrompt
    PredCake -->|"reads"| ModelMaxLen
    PredCake -->|"reads"| ModelPathMap
    PredCake -->|"reads"| ModelTau
    PredCake -->|"initializes"| ModelAdapter
    PredCake -->|"loads model via HF API"| HF
    ModelAdapter -->|"applies patches to"| HF
    ModelAdapter -->|"registers eviction hooks"| CacheManager
    HF -->|"invokes eviction logic"| CacheManager
    CacheManager -->|"uses utils for metrics"| Utils
    HF -->|"uses tensor ops"| Torch
    HF -->|"generates tokens"| Predictions
    Predictions -->|"ingested by"| Eval
    Eval -->|"calls metrics in"| MetricsScript
    MetricsScript -->|"writes"| EvalResults

    %% Click Events
    click PredCake "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/pred_cake.py"
    click Eval "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/eval.py"
    click DSMaxLen "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/config/dataset2maxlen.json"
    click DSPrompt "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/config/dataset2prompt.json"
    click ModelMaxLen "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/config/model2maxlen.json"
    click ModelPathMap "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/config/model2path.json"
    click ModelTau "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/config/model2tau.json"
    click CacheManager "https://github.com/antgroup/cakekv/blob/main/cake/cake_cache.py"
    click ModelAdapter "https://github.com/antgroup/cakekv/blob/main/cake/monkeypatch.py"
    click LLamaAdapter "https://github.com/antgroup/cakekv/blob/main/cake/model/modify_llama.py"
    click MistralAdapter "https://github.com/antgroup/cakekv/blob/main/cake/model/modify_mistral.py"
    click Qwen2Adapter "https://github.com/antgroup/cakekv/blob/main/cake/model/modify_qwen2.py"
    click Utils "https://github.com/antgroup/cakekv/blob/main/cake/utils.py"
    click MetricsScript "https://github.com/antgroup/cakekv/blob/main/experiments/LongBench/metrics.py"

    %% Styles
    classDef experiment fill:#f9f,stroke:#333,stroke-width:1px
    classDef core fill:#9f9,stroke:#333,stroke-width:1px
    classDef config fill:#ff9,stroke:#333,stroke-width:1px
    classDef external fill:#9ff,stroke:#333,stroke-width:1px
    classDef output fill:#f99,stroke:#333,stroke-width:1px
```
