How safeailab/eagle works
1. What this repo is
- EAGLE is a Python package for accelerating autoregressive LLM decoding through speculative draft generation.
- The repository is organized around three main flows: interactive inference, offline evaluation, and training of the EAGLE draft model.
2. Core inference boundary
- The central runtime object is eagle.model.ea_model.EaModel.
- Inference starts from a base Hugging Face LLM plus an EAGLE checkpoint, then runs eagenerate(...) instead of vanilla generate(...).
- The key architectural idea is that the base model and the EAGLE draft component cooperate during decoding to propose multiple future tokens while preserving output consistency with the base model.
3. Model adapter layer
- eagle/model/modeling_*_kv.py contains model-specific wrappers for supported backbones such as LLaMA, Mixtral, Qwen2, and Qwen3.
- These files adapt the underlying Transformer forward pass to use a preallocated KV cache, which is important for speed and for supporting multi-GPU sharding via device_map="auto".
- kv_cache.py, utils.py, and utils_c.py support the low-level runtime path.
4. Draft model / tree logic
- cnets.py, choices.py, configs.py, and ea_model.py together define the EAGLE draft architecture and how candidate tokens/features are selected.
- The repo supports both EAGLE-2 and EAGLE-3 behavior, with the current main branch defaulting to EAGLE-3.
- The draft model is relatively small compared with the base LLM, and the architecture is tuned to generate token trees rather than a single next-token guess.
5. User-facing application
- eagle/application/webui.py is the interactive entry point shown in the README.
- It loads the base model and EAGLE weights, configures the draft token budget (total_token), and exposes a browser UI for prompt/response testing.
- This layer is thin; it delegates all actual decoding to the model package.
6. Evaluation subsystem
- eagle/evaluation/ contains benchmark runners for baseline decoding and EAGLE-accelerated decoding.
- Scripts like gen_ea_answer_* and gen_baseline_answer_* generate .jsonl outputs containing completions plus timing data.
- speed.py then compares wall-clock performance, while alpha.py supports analysis of decoding behavior/acceptance dynamics.
7. Benchmark data assets
- eagle/data/ packages prompt sets and judge/reference artifacts for tasks such as MT-Bench, GSM8K, HumanEval, Alpaca, QA, and summarization.
- MT-Bench includes both reference answers and model judgments, indicating the repo’s evaluation path is not just generation but also comparison/scoring.
- These assets feed the evaluation scripts and keep benchmark prompting consistent.
8. Training subsystem
- eagle/train/ contains the main training entry points plus model-specific config files for Vicuna, LLaMA 2, Mixtral, and related setups.
- main.py and main_deepspeed.py are the primary trainers; ds_config.json shows DeepSpeed is the scaling mechanism for distributed training.
- The training target is the EAGLE draft model, not the full base LLM.
9. EAGLE-3 training variant
- eagle/traineagle3/ is a separate, newer training stack for EAGLE-3.
- It has its own main.py, default_config.yaml, ds_config.json, and LLaMA KV-model implementation, indicating EAGLE-3 needed a distinct training path rather than a small tweak to the original trainer.
- This subtree is effectively a versioned training pipeline for the newer architecture.
10. Debugging / experimental sandbox
- eagle/testbug/ is a parallel copy of model code and a test harness.
- Its presence suggests the maintainers use it as a controlled sandbox for reproducing bugs and validating model changes without disturbing the main implementation.
- It is not part of the primary production path.
11. Versioning and supported backbones
- The README and code structure show the repo is centered on LLaMA-family, Vicuna, Mixtral, Qwen2, Qwen3, and some newer custom model families.
- Backward compatibility exists through branch/version differences: the main branch focuses on EAGLE-3/EAGLE-2, while older EAGLE-1 lives elsewhere.
- The architecture diagram should show “base LLM family” as a pluggable boundary, with model-specific adapters beneath the common EAGLE runtime.
12. Packaging and runtime dependencies
- This is a Python package (setup.py) with standard dependency files for CUDA/NVIDIA and ROCm environments.
- DeepSpeed is the main training infrastructure; Hugging Face model loading and chat templates are the main external model-serving dependencies.
- The repo is therefore best diagrammed as a library plus scripts, not a standalone service with persistent backend state.
Main entry points: ea_model.py EaModel · modeling_llama_kv.py KV Adapters · gen_ea_answer_llama2chat.py Eval Generators · main.py Train Main
Components 22
Runtime
Core inference and model-adapter path for speculative decoding.
- EaModel · Central EAGLE inference object that coordinates base LLM decoding with speculative draft generation.
eagle/model/ea_model.py - KV Adapters · Model-specific wrappers that adapt supported backbones to the EAGLE runtime with preallocated KV cache support.
eagle/model/modeling_llama_kv.py - Draft Nets · Candidate-token/tree network logic for EAGLE draft generation.
eagle/model/cnets.py - Token Choices · Selection and branching logic used to form and rank speculative candidates.
eagle/model/choices.py - Draft Configs · Configuration definitions that parameterize EAGLE draft behavior and variants.
eagle/model/configs.py - KV Cache · Preallocated cache machinery used to speed decoding and support sharded execution.
eagle/model/kv_cache.py - Model Utils · Shared Python helpers for the low-level model execution path.
eagle/model/utils.py - C Utils · Compiled helper path for performance-critical runtime operations.
eagle/model/utils_c.py
Apps & Eval
Interactive UI and benchmark generation/analysis flows.
- Web UI · Browser-facing entry point that loads models, sets token budget, and delegates decoding to EaModel.
eagle/application/webui.py - Eval Generators · Scripts that generate baseline and EAGLE-accelerated answers for different model families and tasks.
eagle/evaluation/gen_ea_answer_llama2chat.py - Speed · Wall-clock performance comparison between baseline and EAGLE decoding.
eagle/evaluation/speed.py - Alpha · Analysis of acceptance and decoding dynamics for speculative generation.
eagle/evaluation/alpha.py
Training
Draft-model training pipelines, including the EAGLE-3 variant.
- Train Main · Primary training entry point for the EAGLE draft model.
eagle/train/main.py - DeepSpeed Train · Distributed training entry point wired to DeepSpeed.
eagle/train/main_deepspeed.py - Train Configs · Model-family training configurations for Vicuna, LLaMA 2, and Mixtral setups.
eagle/train/vicuna_7B_config.json - DS Config · DeepSpeed configuration used by the training stack.
eagle/train/ds_config.json - EAGLE-3 Train · Separate training pipeline for the newer EAGLE-3 architecture.
eagle/traineagle3/main.py - E3 KV Model · EAGLE-3 LLaMA KV-cache model implementation.
eagle/traineagle3/modeling_llama_kv.py
Assets
Packaged benchmark prompts and reference/judgment artifacts.
- Bench Data · Packaged benchmark question sets and judge/reference artifacts used by evaluation scripts.
eagle/data/mt_bench/question.jsonl - Judge Data · Reference answers and model judgments for benchmark comparison and scoring.
eagle/data/mt_bench/reference_answer/gpt-4.jsonl
Sandbox
Experimental and bug-reproduction code path.
- Sandbox Model · Parallel experimental copy of the core model path for bug reproduction and validation.
eagle/testbug/model/ea_model.py - Sandbox Tools · Bug-reproduction and visualization utilities for the sandbox workflow.
eagle/testbug/testbbug.py
Connections 23
- Web UI → EaModel: uses · Interactive UI delegates all decoding to the core runtime.
- EaModel → Draft Nets: orchestrates · EaModel coordinates speculative draft-tree construction.
- EaModel → Token Choices: selects · Core runtime uses candidate-selection logic to choose draft continuations.
- EaModel → Draft Configs: configures · Model behavior is parameterized by EAGLE architecture configs.
- EaModel → KV Cache: uses · Decoding path relies on preallocated KV cache.
- KV Adapters → KV Cache: adapts to · Backbone wrappers are built around the shared cache strategy.
- KV Adapters → Model Utils: shares · Adapters reuse common runtime helpers.
- KV Adapters → C Utils: accelerates · Wrappers depend on compiled helpers for critical operations.
- KV Adapters → EaModel: served by · Supported backbones plug into the EAGLE inference object.
- Web UI → KV Adapters: loads · UI loads a base model backbone plus EAGLE weights.
- Eval Generators → EaModel: benchmarks · Evaluation scripts invoke EAGLE decoding for answer generation.
- Eval Generators → Bench Data: reads · Benchmark runners consume packaged prompt sets.
- Eval Generators → Judge Data: compares · Some evaluation flows use reference/judgment artifacts.
- Speed → Eval Generators: measures · Speed analysis compares generated outputs and timings.
- Alpha → Eval Generators: analyzes · Alpha analysis consumes decoding outputs to study acceptance behavior.
- Train Main → Draft Nets: trains · Primary trainer fits the draft network.
- DeepSpeed Train → Train Main: scales · DeepSpeed entry provides distributed scaling for training.
- DeepSpeed Train → DS Config: uses · Launcher configuration controls distributed execution.
- Train Main → Train Configs: loads · Training uses model-family configuration files.
- EAGLE-3 Train → E3 KV Model: uses · EAGLE-3 trainer is paired with its own KV model implementation.
- EAGLE-3 Train → DS Config: uses · EAGLE-3 training also relies on DeepSpeed config.
- Sandbox Tools → Sandbox Model: inspects · Debug harness exercises the experimental model copy.
- Sandbox Model → EaModel: mirrors · Sandbox path tracks the main runtime architecture for debugging.