# bytedance-seed/flexprefill architecture

> Architecture diagram of the GitHub repository bytedance-seed/flexprefill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance-seed/flexprefill
- Repository: https://github.com/bytedance-seed/flexprefill

## Overview

##### Project type and purpose
- This is a research‐oriented Python library and toolchain (“FlexPrefill”) that implements a novel sparse attention mechanism for long‐sequence inference in LLMs, plus accompanying benchmark/experiment scripts.
- Core purpose: provide Triton‐based kernels and PyTorch/vLLM integration functions to patch Hugging Face models (LLaMA, Qwen2, ChatGLM4, Yi, etc.) with “flex prefill” sparse attention, and benchmark its performance on RULER and InfiniteBench datasets.

##### File structure
- Top-level:
- flex_prefill/        ← Core library package
- experiments/         ← End-to-end benchmark pipelines and configs
- tests/               ← Unit/integration tests
- setup.py, install.sh ← Packaging & install scripts
- extra_requirements.txt ← Optional deps for experiments
- flex_prefill/ substructure:
- modules/             ← Model-specific attention implementations (GLM, llama, qwen2)
- ops/                 ← Low-level Triton kernel wrappers and JSON configs for Minference
- __init__.py          ← Exposes patch_model, flex_prefill_attention APIs
- experiments/ substructure:
- benchmark/           ← YAML configs, data prep, evaluation code for InfiniteBench & RULER
- scripts/             ← Shell scripts to run flex_prefill vs. default vs. other modes
- download_data.sh, download_model.sh
- Tests: simple LLm inference timing and kernel benchmarks.

##### README for additional insights
- Describes dependencies: torch, triton, transformers, vllm
- Shows how to call flex_prefill_attention directly and how to patch models.
- Outlines experiment setup and scripts.

##### Components
a. Main components
- Core Library:
- API Layer: patch_model, flex_prefill_attention
- Modules: model‐specific attention implementations
- Ops: Triton kernels, Minference configs
- Model Hosts:
- PyTorch/HuggingFace Transformers
- vLLM runtime
- Experiment Pipeline:
- Dataset Download & Prep (InfiniteBench, RULER)
- Benchmark Runner Scripts
- Results Storage
- External Dependencies: Triton, FlashAttention, GPU Hardware
- Tests & CI
b. Relationships & interactions
- patch_model → hooks into Transformers/vLLM internals → redirects attention calls to flex_prefill.ops
- flex_prefill_attention API → invokes flex_prefill.modules.* → uses Triton kernels via flex_prefill.ops
- experiments/scripts → orchestrate download_model.sh, download_dataset.sh → invoke Python benchmark entrypoints (run_ruler.py, run_infinitebench.py) → generate results
- tests → directly call flex_prefill_attention or patched models → validate correctness/performance
c. Architectural patterns / design principles
- Plugin/Adapter pattern: patch_model adapts existing models without forking core code
- Modular separation:
- High-level API vs. low-level kernels
- Per-model implementations in distinct modules
- Experiment orchestration decoupled from core library
- Config‐driven benchmarks: YAML files parameterize runs without code changes
d. Key technologies, frameworks, libraries
- Python packaging (setup.py)
- PyTorch, Triton, flash_attn
- Hugging Face Transformers, vLLM
- Shell scripting for experiments
- YAML for experiment configs
- Triton kernels for attention

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User & Tests Layer
    subgraph "User Code & CI" 
        direction TB
        UC["User Code / Scripts"]:::external
        T1["tests/test_llm.py"]:::core
        T2["tests/kernel_benchmark.py"]:::core
        TD["tests/example_data.json"]:::storage
        UC -->|calls| T1
        UC -->|calls| T2
    end

    %% Core Library Layer
    subgraph "Core Library" 
        direction TB
        subgraph "API Layer"
            direction TB
            API1["patch_model()"]:::core
            API2["flex_prefill_attention()"]:::core
        end
        subgraph "Model-Specific Modules"
            direction TB
            M1["glm/"]:::core
            M2["llama/"]:::core
            M3["qwen2/"]:::core
        end
        subgraph "Operations (Ops)"
            direction TB
            O1["minfer_attention.py"]:::core
            O2["config/*.json"]:::core
            O3["streaming_llm_attention.py"]:::core
            O4["vertical_slash_attention.py"]:::core
        end

        API1 -->|invokes| API2
        API2 -->|uses| M1
        API2 -->|uses| M2
        API2 -->|uses| M3
        API2 -->|calls| O1
        API2 -->|calls| O3
        API2 -->|calls| O4
        O1 -->|reads| O2
    end

    %% Host Frameworks & Hardware
    subgraph "Host & Hardware" 
        direction TB
        HF1["HuggingFace Transformers"]:::external
        HF2["vLLM Runtime"]:::external
        GPU["GPU Hardware"]:::external
        HF1 -->|patched by| API1
        HF2 -->|patched by| API1
        O1 -->|executes on| GPU
        O3 -->|executes on| GPU
        O4 -->|executes on| GPU
    end

    %% Experiment Pipeline
    subgraph "Experiment Pipeline" 
        direction TB
        subgraph "Data Preparation & Config"
            direction TB
            D1["download_data.sh"]:::experiment
            D2["infinitebench/download_dataset.sh"]:::experiment
            D3["ruler/download_dataset.sh"]:::experiment
            CFG1["infinitebench/*.yaml"]:::experiment
            CFG2["ruler/data/*"]:::experiment
        end
        subgraph "Benchmark Runners"
            direction TB
            R1["run_infinitebench.py"]:::experiment
            R2["run_ruler.py"]:::experiment
        end
        subgraph "Scripts"
            direction TB
            S1["scripts/flex_prefill/"]:::experiment
            S2["scripts/full/"]:::experiment
            S3["scripts/minfer/"]:::experiment
            S4["scripts/streaming_llm/"]:::experiment
        end
        subgraph "Results Storage"
            direction TB
            RS["Results/"]:::storage
        end

        D1 --> D2
        D1 --> D3
        D2 -->|outputs| CFG1
        D3 -->|outputs| CFG2
        CFG1 --> R1
        CFG2 --> R2
        S1 --> R1
        S2 --> R1
        S3 --> R1
        S4 --> R2
        R1 -->|writes| RS
        R2 -->|writes| RS
    end

    %% Click Events
    click API1 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/__init__.py"
    click API1 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/modules/patch.py"
    click API2 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/ops/flex_prefill_attention.py"
    click M1 "https://github.com/bytedance-seed/flexprefill/tree/main/flex_prefill/modules/glm/"
    click M2 "https://github.com/bytedance-seed/flexprefill/tree/main/flex_prefill/modules/llama/"
    click M3 "https://github.com/bytedance-seed/flexprefill/tree/main/flex_prefill/modules/qwen2/"
    click O1 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/ops/minfer/minfer_attention.py"
    click O2 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/ops/minfer/config/*.json"
    click O3 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/ops/streaming_llm_attention.py"
    click O4 "https://github.com/bytedance-seed/flexprefill/blob/main/flex_prefill/ops/vertical_slash_attention.py"
    click D1 "https://github.com/bytedance-seed/flexprefill/blob/main/experiments/download_data.sh"
    click D2 "https://github.com/bytedance-seed/flexprefill/blob/main/experiments/benchmark/infinitebench/download_dataset.sh"
    click D3 "https://github.com/bytedance-seed/flexprefill/blob/main/experiments/benchmark/ruler/download_dataset.sh"
    click CFG1 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/benchmark/infinitebench/"
    click CFG2 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/benchmark/ruler/data/"
    click R1 "https://github.com/bytedance-seed/flexprefill/blob/main/experiments/benchmark/run_infinitebench.py"
    click R2 "https://github.com/bytedance-seed/flexprefill/blob/main/experiments/benchmark/run_ruler.py"
    click S1 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/scripts/flex_prefill/"
    click S2 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/scripts/full/"
    click S3 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/scripts/minfer/"
    click S4 "https://github.com/bytedance-seed/flexprefill/tree/main/experiments/scripts/streaming_llm/"
    click T1 "https://github.com/bytedance-seed/flexprefill/blob/main/tests/test_llm.py"
    click T2 "https://github.com/bytedance-seed/flexprefill/blob/main/tests/kernel_benchmark.py"

    %% Styles
    classDef core fill:#cce5ff,stroke:#005cbf,color:#002752
    classDef experiment fill:#d4edda,stroke:#155724,color:#0b2e13
    classDef external fill:#f8d7da,stroke:#721c24,color:#491217
    classDef storage fill:#fff3cd,stroke:#856404,color:#533f03
```
