# lucidrains/speculative-decoding architecture

> Architecture diagram of the GitHub repository lucidrains/speculative-decoding, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucidrains/speculative-decoding
- Repository: https://github.com/lucidrains/speculative-decoding

## Overview

##### Project Type and Purpose
- This repository is a research‐oriented Python library and set of training scripts for experimenting with speculative decoding techniques on transformer models.
- It provides implementations of “early exit” and “prophet” speculative decoding strategies, and standalone training scripts to run experiments on a dataset (data/enwik8.gz).
- The ultimate goal is to speed up generation (e.g., in text-to-semantic TTS) by drafting with a small or partial model and verifying with a full model.

##### File Structure Analysis
- Top-level directories/files:
– .github/workflows: CI for Python packaging
– data/: raw dataset (enwik8) and its README
– speculative_decoding/: core Python package (two module files)
– setup.py: package installation script
– train.py, train_early_exit.py, train_prophet.py: experiment entrypoints
- Patterns and components:
– “speculative_decoding” package holds the reusable decoding algorithms
– “train_*.py” scripts orchestrate data loading, model instantiation, decoding loops, and logging
– Data folder contains compressed text data for language modeling experiments

##### README Insights
- Describes focus on speculative decoding and proprietary ideas (early exit, prophet transformer)
- Mentions external sponsors (StabilityAI, Huggingface) and intended downstream use in Spear-TTS

##### Main Components
a. Data Layer
– Raw dataset (data/enwik8.gz)
– DataLoader module in training scripts
b. Core Library
– SpeculativeDecoding class (in speculative_decoding/speculative_decoding.py)
– EarlyExit subclass/extension (speculative_decoding_with_prophet.py or analogous)
– Prophet decoding strategy (speculative_decoding_with_prophet.py)
c. Models
– Small “draft” transformer (configurable number of layers)
– Full “verify” transformer (complete model)
d. Training Orchestration
– Entry scripts (train.py, train_early_exit.py, train_prophet.py)
– Instantiates data loader, draft and verify models, decoding module, optimizer
– Logging/tracking (printouts or external service like wandb)
e. External Dependencies
– PyTorch (torch, torch.nn)
– Huggingface Transformers (model definitions, tokenizers)
– Optional: Weights & Biases for experiment tracking

##### Relationships and Interactions
- DataLoader → supplies token batches to Training Orchestrator
- Training Orchestrator → uses SpeculativeDecoding module:
– calls draft_model.forward() to generate draft tokens
– routes accepted drafts to full_model.forward() for verification
– updates caching (hidden layers) for early‐exit strategy
- Optimizer updates model weights, metrics sent to logging backend
- Package module is imported by all train_*.py scripts, providing a plug-in interface for different decoding strategies

##### Architectural Patterns and Principles
- Separation of concerns:
– Core decoding logic encapsulated in a library (speculative_decoding/)
– Scripts handle experiment configuration and orchestration
- Plugin pattern: each decoding strategy (standard speculative, early exit, prophet) is a self-contained class
- Reusability: setup.py makes the library installable and importable in other projects (e.g., Spear-TTS)
- Data‐flow emphasis: batch → draft → verify → loss calculation → back-prop

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "Data Layer"
    DLData["enwik8.gz"]:::data
    DLDoc["data/README.md"]:::data
  end

  subgraph "Core Library"
    SDinit["__init__.py"]:::library
    SDbase["speculative_decoding.py"]:::library
    SDprophet["speculative_decoding_with_prophet.py"]:::library
    Cache["Cache Buffer"]:::library
  end

  subgraph "Training Orchestration"
    TrainStd["train.py"]:::orchestration
    TrainEE["train_early_exit.py"]:::orchestration
    TrainPH["train_prophet.py"]:::orchestration
    DataLoader["DataLoader"]:::orchestration
    DraftModel["DraftTransformer"]:::model
    SpecModule["SpeculativeDecoding Module"]:::library
    VerifyModel["VerifyTransformer"]:::model
    Optim["Optimizer"]:::orchestration
    Log["Logger (wandb/print)"]:::orchestration
  end

  subgraph "Project Config & CI"
    Setup["setup.py"]:::orchestration
    CI["python-publish.yml"]:::external
    MainReadme["README.md"]:::external
    License["LICENSE"]:::external
    Ignore[".gitignore"]:::external
  end

  subgraph "External Dependencies"
    Torch["PyTorch"]:::external
    HF["Huggingface Transformers"]:::external
    WB["Weights & Biases"]:::external
  end

  DLData -->|"loads data"| DataLoader
  DLDoc --> DataLoader
  DataLoader -->|"batch tokens"| DraftModel
  DraftModel -->|"draft tokens"| SpecModule
  SpecModule -->|"accepted drafts"| VerifyModel
  SpecModule -->|"cache updates"| Cache
  VerifyModel -->|"verified tokens"| Optim
  Optim -->|"gradients"| DraftModel
  Optim -->|"gradients"| VerifyModel
  Optim --> Log

  TrainStd --> DataLoader
  TrainStd --> SpecModule
  TrainStd --> DraftModel
  TrainStd --> VerifyModel
  TrainStd --> Optim
  TrainStd --> Log

  TrainEE --> TrainStd
  TrainPH --> TrainStd

  TrainStd --> Torch
  TrainStd --> HF
  Log --> WB

  click DLData "https://github.com/lucidrains/speculative-decoding/blob/main/data/enwik8.gz"
  click DLDoc "https://github.com/lucidrains/speculative-decoding/blob/main/data/README.md"
  click SDbase "https://github.com/lucidrains/speculative-decoding/blob/main/speculative_decoding/speculative_decoding.py"
  click SDprophet "https://github.com/lucidrains/speculative-decoding/blob/main/speculative_decoding/speculative_decoding_with_prophet.py"
  click SDinit "https://github.com/lucidrains/speculative-decoding/blob/main/speculative_decoding/__init__.py"
  click TrainStd "https://github.com/lucidrains/speculative-decoding/blob/main/train.py"
  click TrainEE "https://github.com/lucidrains/speculative-decoding/blob/main/train_early_exit.py"
  click TrainPH "https://github.com/lucidrains/speculative-decoding/blob/main/train_prophet.py"
  click Setup "https://github.com/lucidrains/speculative-decoding/blob/main/setup.py"
  click CI "https://github.com/lucidrains/speculative-decoding/blob/main/.github/workflows/python-publish.yml"
  click MainReadme "https://github.com/lucidrains/speculative-decoding/blob/main/README.md"
  click License "https://github.com/lucidrains/speculative-decoding/tree/main/LICENSE"
  click Ignore "https://github.com/lucidrains/speculative-decoding/blob/main/.gitignore"

  classDef data fill:#D6E9FF,stroke:#4D90FE,color:#000
  classDef model fill:#DFF2BF,stroke:#7DBE00,color:#000
  classDef library fill:#FFF8C6,stroke:#FFD324,color:#000
  classDef orchestration fill:#FFE1B3,stroke:#FF9800,color:#000
  classDef external fill:#F0F0F0,stroke:#AAA,color:#000
```
