# safeailab/eagle architecture

> Architecture diagram of the GitHub repository safeailab/eagle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/safeailab/eagle
- Repository: https://github.com/safeailab/eagle

## Overview

1. **What this repo is**
   - EAGLE is a Python package for accelerating autoregressive LLM decoding through speculative draft generation.
   - The repository is organized around three main flows: interactive inference, offline evaluation, and training of the EAGLE draft model.

2. **Core inference boundary**
   - The central runtime object is `eagle.model.ea_model.EaModel`.
   - Inference starts from a base Hugging Face LLM plus an EAGLE checkpoint, then runs `eagenerate(...)` instead of vanilla `generate(...)`.
   - The key architectural idea is that the base model and the EAGLE draft component cooperate during decoding to propose multiple future tokens while preserving output consistency with the base model.

3. **Model adapter layer**
   - `eagle/model/modeling_*_kv.py` contains model-specific wrappers for supported backbones such as LLaMA, Mixtral, Qwen2, and Qwen3.
   - These files adapt the underlying Transformer forward pass to use a preallocated KV cache, which is important for speed and for supporting multi-GPU sharding via `device_map="auto"`.
   - `kv_cache.py`, `utils.py`, and `utils_c.py` support the low-level runtime path.

4. **Draft model / tree logic**
   - `cnets.py`, `choices.py`, `configs.py`, and `ea_model.py` together define the EAGLE draft architecture and how candidate tokens/features are selected.
   - The repo supports both EAGLE-2 and EAGLE-3 behavior, with the current main branch defaulting to EAGLE-3.
   - The draft model is relatively small compared with the base LLM, and the architecture is tuned to generate token trees rather than a single next-token guess.

5. **User-facing application**
   - `eagle/application/webui.py` is the interactive entry point shown in the README.
   - It loads the base model and EAGLE weights, configures the draft token budget (`total_token`), and exposes a browser UI for prompt/response testing.
   - This layer is thin; it delegates all actual decoding to the model package.

6. **Evaluation subsystem**
   - `eagle/evaluation/` contains benchmark runners for baseline decoding and EAGLE-accelerated decoding.
   - Scripts like `gen_ea_answer_*` and `gen_baseline_answer_*` generate `.jsonl` outputs containing completions plus timing data.
   - `speed.py` then compares wall-clock performance, while `alpha.py` supports analysis of decoding behavior/acceptance dynamics.

7. **Benchmark data assets**
   - `eagle/data/` packages prompt sets and judge/reference artifacts for tasks such as MT-Bench, GSM8K, HumanEval, Alpaca, QA, and summarization.
   - MT-Bench includes both reference answers and model judgments, indicating the repo’s evaluation path is not just generation but also comparison/scoring.
   - These assets feed the evaluation scripts and keep benchmark prompting consistent.

8. **Training subsystem**
   - `eagle/train/` contains the main training entry points plus model-specific config files for Vicuna, LLaMA 2, Mixtral, and related setups.
   - `main.py` and `main_deepspeed.py` are the primary trainers; `ds_config.json` shows DeepSpeed is the scaling mechanism for distributed training.
   - The training target is the EAGLE draft model, not the full base LLM.

9. **EAGLE-3 training variant**
   - `eagle/traineagle3/` is a separate, newer training stack for EAGLE-3.
   - It has its own `main.py`, `default_config.yaml`, `ds_config.json`, and LLaMA KV-model implementation, indicating EAGLE-3 needed a distinct training path rather than a small tweak to the original trainer.
   - This subtree is effectively a versioned training pipeline for the newer architecture.

10. **Debugging / experimental sandbox**
   - `eagle/testbug/` is a parallel copy of model code and a test harness.
   - Its presence suggests the maintainers use it as a controlled sandbox for reproducing bugs and validating model changes without disturbing the main implementation.
   - It is not part of the primary production path.

11. **Versioning and supported backbones**
   - The README and code structure show the repo is centered on LLaMA-family, Vicuna, Mixtral, Qwen2, Qwen3, and some newer custom model families.
   - Backward compatibility exists through branch/version differences: the main branch focuses on EAGLE-3/EAGLE-2, while older EAGLE-1 lives elsewhere.
   - The architecture diagram should show “base LLM family” as a pluggable boundary, with model-specific adapters beneath the common EAGLE runtime.

12. **Packaging and runtime dependencies**
   - This is a Python package (`setup.py`) with standard dependency files for CUDA/NVIDIA and ROCm environments.
   - DeepSpeed is the main training infrastructure; Hugging Face model loading and chat templates are the main external model-serving dependencies.
   - The repo is therefore best diagrammed as a library plus scripts, not a standalone service with persistent backend state.

Main entry points: [`eagle/model/ea_model.py`](https://github.com/safeailab/eagle/blob/main/eagle/model/ea_model.py) (EaModel), [`eagle/model/modeling_llama_kv.py`](https://github.com/safeailab/eagle/blob/main/eagle/model/modeling_llama_kv.py) (KV Adapters), [`eagle/evaluation/gen_ea_answer_llama2chat.py`](https://github.com/safeailab/eagle/blob/main/eagle/evaluation/gen_ea_answer_llama2chat.py) (Eval Generators), [`eagle/train/main.py`](https://github.com/safeailab/eagle/blob/main/eagle/train/main.py) (Train Main)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_runtime["Runtime"]
  node_n_ea_model["EaModel<br/>core runtime<br/>[ea_model.py]"]
  node_n_modeling_family["KV Adapters<br/>backbone wrappers"]
  node_n_cnets["Draft Nets<br/>draft model<br/>[cnets.py]"]
  node_n_choices["Token Choices<br/>selection logic<br/>[choices.py]"]
  node_n_configs["Draft Configs<br/>architecture config<br/>[configs.py]"]
  node_n_kv_cache[("KV Cache<br/>runtime cache<br/>[kv_cache.py]")]
  node_n_model_utils["Model Utils<br/>runtime helpers<br/>[utils.py]"]
  node_n_model_utils_c["C Utils<br/>native helpers<br/>[utils_c.py]"]
end

subgraph group_g_apps["Apps & Eval"]
  node_n_webui["Web UI<br/>interactive app<br/>[webui.py]"]
  node_n_eval_gen["Eval Generators<br/>benchmark runners"]
  node_n_speed["Speed<br/>benchmark analysis<br/>[speed.py]"]
  node_n_alpha["Alpha<br/>decoding analysis<br/>[alpha.py]"]
end

subgraph group_g_training["Training"]
  node_n_train_main["Train Main<br/>training entry<br/>[main.py]"]
  node_n_train_deepspeed["DeepSpeed Train<br/>distributed training<br/>[main_deepspeed.py]"]
  node_n_train_configs["Train Configs<br/>model configs"]
  node_n_train_ds["DS Config<br/>launcher config<br/>[ds_config.json]"]
  node_n_e3_main["EAGLE-3 Train<br/>versioned trainer<br/>[main.py]"]
  node_n_e3_model["E3 KV Model<br/>backbone wrapper"]
end

subgraph group_g_assets["Assets"]
  node_n_data_bench["Bench Data<br/>prompt assets<br/>[question.jsonl]"]
  node_n_data_judge["Judge Data<br/>judging assets<br/>[gpt-4.jsonl]"]
end

subgraph group_g_sandbox["Sandbox"]
  node_n_sandbox_model["Sandbox Model<br/>experimental copy<br/>[ea_model.py]"]
  node_n_sandbox_tools["Sandbox Tools<br/>debug harness<br/>[testbbug.py]"]
end

node_n_webui -->|"uses"| node_n_ea_model
node_n_ea_model -->|"orchestrates"| node_n_cnets
node_n_ea_model -->|"selects"| node_n_choices
node_n_ea_model -->|"configures"| node_n_configs
node_n_ea_model -->|"uses"| node_n_kv_cache
node_n_modeling_family -->|"adapts to"| node_n_kv_cache
node_n_modeling_family -->|"shares"| node_n_model_utils
node_n_modeling_family -->|"accelerates"| node_n_model_utils_c
node_n_modeling_family -->|"served by"| node_n_ea_model
node_n_webui -->|"loads"| node_n_modeling_family
node_n_eval_gen -->|"benchmarks"| node_n_ea_model
node_n_eval_gen -->|"reads"| node_n_data_bench
node_n_eval_gen -->|"compares"| node_n_data_judge
node_n_speed -->|"measures"| node_n_eval_gen
node_n_alpha -->|"analyzes"| node_n_eval_gen
node_n_train_main -->|"trains"| node_n_cnets
node_n_train_deepspeed -->|"scales"| node_n_train_main
node_n_train_deepspeed -->|"uses"| node_n_train_ds
node_n_train_main -->|"loads"| node_n_train_configs
node_n_e3_main -->|"uses"| node_n_e3_model
node_n_e3_main -->|"uses"| node_n_train_ds
node_n_sandbox_tools -->|"inspects"| node_n_sandbox_model
node_n_sandbox_model -.->|"mirrors"| node_n_ea_model

click node_n_ea_model "https://github.com/safeailab/eagle/blob/main/eagle/model/ea_model.py"
click node_n_modeling_family "https://github.com/safeailab/eagle/blob/main/eagle/model/modeling_llama_kv.py"
click node_n_cnets "https://github.com/safeailab/eagle/blob/main/eagle/model/cnets.py"
click node_n_choices "https://github.com/safeailab/eagle/blob/main/eagle/model/choices.py"
click node_n_configs "https://github.com/safeailab/eagle/blob/main/eagle/model/configs.py"
click node_n_kv_cache "https://github.com/safeailab/eagle/blob/main/eagle/model/kv_cache.py"
click node_n_model_utils "https://github.com/safeailab/eagle/blob/main/eagle/model/utils.py"
click node_n_model_utils_c "https://github.com/safeailab/eagle/blob/main/eagle/model/utils_c.py"
click node_n_webui "https://github.com/safeailab/eagle/blob/main/eagle/application/webui.py"
click node_n_eval_gen "https://github.com/safeailab/eagle/blob/main/eagle/evaluation/gen_ea_answer_llama2chat.py"
click node_n_speed "https://github.com/safeailab/eagle/blob/main/eagle/evaluation/speed.py"
click node_n_alpha "https://github.com/safeailab/eagle/blob/main/eagle/evaluation/alpha.py"
click node_n_train_main "https://github.com/safeailab/eagle/blob/main/eagle/train/main.py"
click node_n_train_deepspeed "https://github.com/safeailab/eagle/blob/main/eagle/train/main_deepspeed.py"
click node_n_train_configs "https://github.com/safeailab/eagle/blob/main/eagle/train/vicuna_7B_config.json"
click node_n_train_ds "https://github.com/safeailab/eagle/blob/main/eagle/train/ds_config.json"
click node_n_e3_main "https://github.com/safeailab/eagle/blob/main/eagle/traineagle3/main.py"
click node_n_e3_model "https://github.com/safeailab/eagle/blob/main/eagle/traineagle3/modeling_llama_kv.py"
click node_n_data_bench "https://github.com/safeailab/eagle/blob/main/eagle/data/mt_bench/question.jsonl"
click node_n_data_judge "https://github.com/safeailab/eagle/blob/main/eagle/data/mt_bench/reference_answer/gpt-4.jsonl"
click node_n_sandbox_model "https://github.com/safeailab/eagle/blob/main/eagle/testbug/model/ea_model.py"
click node_n_sandbox_tools "https://github.com/safeailab/eagle/blob/main/eagle/testbug/testbbug.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_ea_model,node_n_modeling_family,node_n_cnets,node_n_choices,node_n_configs,node_n_kv_cache,node_n_model_utils,node_n_model_utils_c toneBlue
class node_n_webui,node_n_eval_gen,node_n_speed,node_n_alpha toneAmber
class node_n_train_main,node_n_train_deepspeed,node_n_train_configs,node_n_train_ds,node_n_e3_main,node_n_e3_model toneMint
class node_n_data_bench,node_n_data_judge toneRose
class node_n_sandbox_model,node_n_sandbox_tools toneIndigo
```

## Components

### Runtime

Core inference and model-adapter path for speculative decoding.

- [EaModel](https://github.com/safeailab/eagle/blob/main/eagle/model/ea_model.py): core runtime · Central EAGLE inference object that coordinates base LLM decoding with speculative draft generation. · `eagle/model/ea_model.py`
- [KV Adapters](https://github.com/safeailab/eagle/blob/main/eagle/model/modeling_llama_kv.py): backbone wrappers · Model-specific wrappers that adapt supported backbones to the EAGLE runtime with preallocated KV cache support. · `eagle/model/modeling_llama_kv.py`
- [Draft Nets](https://github.com/safeailab/eagle/blob/main/eagle/model/cnets.py): draft model · Candidate-token/tree network logic for EAGLE draft generation. · `eagle/model/cnets.py`
- [Token Choices](https://github.com/safeailab/eagle/blob/main/eagle/model/choices.py): selection logic · Selection and branching logic used to form and rank speculative candidates. · `eagle/model/choices.py`
- [Draft Configs](https://github.com/safeailab/eagle/blob/main/eagle/model/configs.py): architecture config · Configuration definitions that parameterize EAGLE draft behavior and variants. · `eagle/model/configs.py`
- [KV Cache](https://github.com/safeailab/eagle/blob/main/eagle/model/kv_cache.py): runtime cache · Preallocated cache machinery used to speed decoding and support sharded execution. · `eagle/model/kv_cache.py`
- [Model Utils](https://github.com/safeailab/eagle/blob/main/eagle/model/utils.py): runtime helpers · Shared Python helpers for the low-level model execution path. · `eagle/model/utils.py`
- [C Utils](https://github.com/safeailab/eagle/blob/main/eagle/model/utils_c.py): native helpers · Compiled helper path for performance-critical runtime operations. · `eagle/model/utils_c.py`

### Apps & Eval

Interactive UI and benchmark generation/analysis flows.

- [Web UI](https://github.com/safeailab/eagle/blob/main/eagle/application/webui.py): interactive app · Browser-facing entry point that loads models, sets token budget, and delegates decoding to EaModel. · `eagle/application/webui.py`
- [Eval Generators](https://github.com/safeailab/eagle/blob/main/eagle/evaluation/gen_ea_answer_llama2chat.py): benchmark runners · Scripts that generate baseline and EAGLE-accelerated answers for different model families and tasks. · `eagle/evaluation/gen_ea_answer_llama2chat.py`
- [Speed](https://github.com/safeailab/eagle/blob/main/eagle/evaluation/speed.py): benchmark analysis · Wall-clock performance comparison between baseline and EAGLE decoding. · `eagle/evaluation/speed.py`
- [Alpha](https://github.com/safeailab/eagle/blob/main/eagle/evaluation/alpha.py): decoding analysis · Analysis of acceptance and decoding dynamics for speculative generation. · `eagle/evaluation/alpha.py`

### Training

Draft-model training pipelines, including the EAGLE-3 variant.

- [Train Main](https://github.com/safeailab/eagle/blob/main/eagle/train/main.py): training entry · Primary training entry point for the EAGLE draft model. · `eagle/train/main.py`
- [DeepSpeed Train](https://github.com/safeailab/eagle/blob/main/eagle/train/main_deepspeed.py): distributed training · Distributed training entry point wired to DeepSpeed. · `eagle/train/main_deepspeed.py`
- [Train Configs](https://github.com/safeailab/eagle/blob/main/eagle/train/vicuna_7B_config.json): model configs · Model-family training configurations for Vicuna, LLaMA 2, and Mixtral setups. · `eagle/train/vicuna_7B_config.json`
- [DS Config](https://github.com/safeailab/eagle/blob/main/eagle/train/ds_config.json): launcher config · DeepSpeed configuration used by the training stack. · `eagle/train/ds_config.json`
- [EAGLE-3 Train](https://github.com/safeailab/eagle/blob/main/eagle/traineagle3/main.py): versioned trainer · Separate training pipeline for the newer EAGLE-3 architecture. · `eagle/traineagle3/main.py`
- [E3 KV Model](https://github.com/safeailab/eagle/blob/main/eagle/traineagle3/modeling_llama_kv.py): backbone wrapper · EAGLE-3 LLaMA KV-cache model implementation. · `eagle/traineagle3/modeling_llama_kv.py`

### Assets

Packaged benchmark prompts and reference/judgment artifacts.

- [Bench Data](https://github.com/safeailab/eagle/blob/main/eagle/data/mt_bench/question.jsonl): prompt assets · Packaged benchmark question sets and judge/reference artifacts used by evaluation scripts. · `eagle/data/mt_bench/question.jsonl`
- [Judge Data](https://github.com/safeailab/eagle/blob/main/eagle/data/mt_bench/reference_answer/gpt-4.jsonl): judging assets · Reference answers and model judgments for benchmark comparison and scoring. · `eagle/data/mt_bench/reference_answer/gpt-4.jsonl`

### Sandbox

Experimental and bug-reproduction code path.

- [Sandbox Model](https://github.com/safeailab/eagle/blob/main/eagle/testbug/model/ea_model.py): experimental copy · Parallel experimental copy of the core model path for bug reproduction and validation. · `eagle/testbug/model/ea_model.py`
- [Sandbox Tools](https://github.com/safeailab/eagle/blob/main/eagle/testbug/testbbug.py): debug harness · Bug-reproduction and visualization utilities for the sandbox workflow. · `eagle/testbug/testbbug.py`

## Connections

- Web UI → EaModel: uses · Interactive UI delegates all decoding to the core runtime.
- EaModel → Draft Nets: orchestrates · EaModel coordinates speculative draft-tree construction.
- EaModel → Token Choices: selects · Core runtime uses candidate-selection logic to choose draft continuations.
- EaModel → Draft Configs: configures · Model behavior is parameterized by EAGLE architecture configs.
- EaModel → KV Cache: uses · Decoding path relies on preallocated KV cache.
- KV Adapters → KV Cache: adapts to · Backbone wrappers are built around the shared cache strategy.
- KV Adapters → Model Utils: shares · Adapters reuse common runtime helpers.
- KV Adapters → C Utils: accelerates · Wrappers depend on compiled helpers for critical operations.
- KV Adapters → EaModel: served by · Supported backbones plug into the EAGLE inference object.
- Web UI → KV Adapters: loads · UI loads a base model backbone plus EAGLE weights.
- Eval Generators → EaModel: benchmarks · Evaluation scripts invoke EAGLE decoding for answer generation.
- Eval Generators → Bench Data: reads · Benchmark runners consume packaged prompt sets.
- Eval Generators → Judge Data: compares · Some evaluation flows use reference/judgment artifacts.
- Speed → Eval Generators: measures · Speed analysis compares generated outputs and timings.
- Alpha → Eval Generators: analyzes · Alpha analysis consumes decoding outputs to study acceptance behavior.
- Train Main → Draft Nets: trains · Primary trainer fits the draft network.
- DeepSpeed Train → Train Main: scales · DeepSpeed entry provides distributed scaling for training.
- DeepSpeed Train → DS Config: uses · Launcher configuration controls distributed execution.
- Train Main → Train Configs: loads · Training uses model-family configuration files.
- EAGLE-3 Train → E3 KV Model: uses · EAGLE-3 trainer is paired with its own KV model implementation.
- EAGLE-3 Train → DS Config: uses · EAGLE-3 training also relies on DeepSpeed config.
- Sandbox Tools → Sandbox Model: inspects · Debug harness exercises the experimental model copy.
- Sandbox Model → EaModel: mirrors · Sandbox path tracks the main runtime architecture for debugging.
