How bytedance-seed/flexprefill works
Project type and purpose
• This is a research‐oriented Python library and toolchain (“FlexPrefill”) that implements a novel sparse attention mechanism for long‐sequence inference in LLMs, plus accompanying benchmark/experiment scripts.
• Core purpose: provide Triton‐based kernels and PyTorch/vLLM integration functions to patch Hugging Face models (LLaMA, Qwen2, ChatGLM4, Yi, etc.) with “flex prefill” sparse attention, and benchmark its performance on RULER and InfiniteBench datasets.
File structure
• Top-level:
• flex_prefill/ ← Core library package
• experiments/ ← End-to-end benchmark pipelines and configs
• tests/ ← Unit/integration tests
• setup.py, install.sh ← Packaging & install scripts
• extra_requirements.txt ← Optional deps for experiments
• flex_prefill/ substructure:
• modules/ ← Model-specific attention implementations (GLM, llama, qwen2)
• ops/ ← Low-level Triton kernel wrappers and JSON configs for Minference
• __init__.py ← Exposes patch_model, flex_prefill_attention APIs
• experiments/ substructure:
• benchmark/ ← YAML configs, data prep, evaluation code for InfiniteBench & RULER
• scripts/ ← Shell scripts to run flex_prefill vs. default vs. other modes
• download_data.sh, download_model.sh
• Tests: simple LLm inference timing and kernel benchmarks.
README for additional insights
• Describes dependencies: torch, triton, transformers, vllm
• Shows how to call flex_prefill_attention directly and how to patch models.
• Outlines experiment setup and scripts.
Components
a. Main components
• Core Library:
• API Layer: patch_model, flex_prefill_attention
• Modules: model‐specific attention implementations
• Ops: Triton kernels, Minference configs
• Model Hosts:
• PyTorch/HuggingFace Transformers
• vLLM runtime
• Experiment Pipeline:
• Dataset Download & Prep (InfiniteBench, RULER)
• Benchmark Runner Scripts
• Results Storage
• External Dependencies: Triton, FlashAttention, GPU Hardware
• Tests & CI
b. Relationships & interactions
• patch_model → hooks into Transformers/vLLM internals → redirects attention calls to flex_prefill.ops
• flex_prefill_attention API → invokes flex_prefill.modules.* → uses Triton kernels via flex_prefill.ops
• experiments/scripts → orchestrate download_model.sh, download_dataset.sh → invoke Python benchmark entrypoints (run_ruler.py, run_infinitebench.py) → generate results
• tests → directly call flex_prefill_attention or patched models → validate correctness/performance
c. Architectural patterns / design principles
• Plugin/Adapter pattern: patch_model adapts existing models without forking core code
• Modular separation:
• High-level API vs. low-level kernels
• Per-model implementations in distinct modules
• Experiment orchestration decoupled from core library
• Config‐driven benchmarks: YAML files parameterize runs without code changes
d. Key technologies, frameworks, libraries
• Python packaging (setup.py)
• PyTorch, Triton, flash_attn
• Hugging Face Transformers, vLLM
• Shell scripting for experiments
• YAML for experiment configs
• Triton kernels for attention