# pytorch-labs/gpt-fast architecture

> Architecture diagram of the GitHub repository pytorch-labs/gpt-fast, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pytorch-labs/gpt-fast
- Repository: https://github.com/pytorch-labs/gpt-fast

## Overview

##### Core Components (Main Layer)
- Model Engine (model.py)
- Text Generation Pipeline (generate.py)
- Tokenizer Component (tokenizer.py)
- Quantization Module (quantize.py)
- Tensor Parallelism Handler (tp.py)

##### Optimization Layers
- Int8/Int4 Quantization Paths
- Speculative Decoding Pipeline
- Tensor Parallelism Distribution

##### Model Support Layer
- LLaMA Family Support
- Mixtral 8x7B Support
- Model Conversion Utilities

##### Central Processing Flow (Top to Bottom)
- [Input Text/Prompt] → [Tokenizer] → [Model Processing] → [Text Generation] → [Output Text]

##### Parallel Tracks (Left Side)
- To the main model processing block

##### Optimization Layer (Right Side)
- Tensor parallelism distribution across GPUs
- Speculative decoding pipeline
- Optimizations to the main processing flow

##### Technical Details
- Memory bandwidth metrics
- GPU distribution patterns
- Model size support ranges
- Quantization grouping options

##### Connection Types
- Double lines: Parallel processing connections

##### The various optimization techniques available

- Performance metrics and their meanings
- Hardware configuration requirements
- Supported model types and sizes

Keep the abstraction level focused on the key architectural components while maintaining enough detail to show the sophisticated optimization techniques. This balance is crucial for understanding both the simplicity and power of the implementation.

Remember to indicate that this is a reference implementation rather than a framework, emphasizing its purpose as a demonstration of PyTorch's native capabilities for high-performance LLM inference.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Legend
        direction TB
        L1[Core Processing]:::core
        L2[Optimization Feature]:::optimization
        L3[Model Support]:::model
        L4[Quantization Path]:::quantization
        L5[Data Flow]
        L6[Optional Path]
        L5 --> L6
        L5 -.- L6
    end

    subgraph Input
        UserPrompt[User Prompt]:::input
    end

    subgraph "Core Processing"
        direction TB
        Tokenizer["Tokenizer Component"]:::core
        ModelEngine["Model Engine"]:::core
        TextGen["Text Generation Pipeline"]:::core
        click Tokenizer "https://github.com/pytorch-labs/gpt-fast/blob/main/tokenizer.py"
        click ModelEngine "https://github.com/pytorch-labs/gpt-fast/blob/main/model.py"
        click TextGen "https://github.com/pytorch-labs/gpt-fast/blob/main/generate.py"
    end

    subgraph "Optimization Layer"
        direction TB
        TensorParallel["Tensor Parallelism\nHandler"]:::optimization
        SpecDecoding["Speculative Decoding\nPipeline"]:::optimization
        click TensorParallel "https://github.com/pytorch-labs/gpt-fast/blob/main/tp.py"
        click SpecDecoding "https://github.com/pytorch-labs/gpt-fast/blob/main/scripts/speculate_34B_bf16.sh"
    end

    subgraph "Quantization Paths"
        direction TB
        Int8["Int8 Quantization"]:::quantization
        Int4["Int4 Quantization"]:::quantization
        GPTQ["GPTQ Module"]:::quantization
        click Int8 "https://github.com/pytorch-labs/gpt-fast/blob/main/quantize.py"
        click Int4 "https://github.com/pytorch-labs/gpt-fast/blob/main/quantize.py"
        click GPTQ "https://github.com/pytorch-labs/gpt-fast/blob/main/GPTQ.py"
    end

    subgraph "Model Support"
        direction TB
        LLaMA["LLaMA Family"]:::model
        Mixtral["Mixtral 8x7B"]:::model
        ModelConversion["Model Conversion\nUtilities"]:::model
        click Mixtral "https://github.com/pytorch-labs/gpt-fast/blob/main/mixtral-moe/model.py"
        click ModelConversion "https://github.com/pytorch-labs/gpt-fast/blob/main/scripts/convert_hf_checkpoint.py"
    end

    subgraph Output
        GeneratedText[Generated Text]:::output
    end

    %% Main Flow
    UserPrompt --> Tokenizer
    Tokenizer --> ModelEngine
    ModelEngine --> TextGen
    TextGen --> GeneratedText

    %% Optimization Connections
    ModelEngine <--> TensorParallel
    TextGen <--> SpecDecoding
    
    %% Quantization Connections
    ModelEngine <-.-> Int8
    ModelEngine <-.-> Int4
    Int4 <--> GPTQ
    
    %% Model Support Connections
    ModelEngine --- LLaMA
    ModelEngine --- Mixtral
    LLaMA & Mixtral --- ModelConversion

    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef optimization fill:#2ecc71,stroke:#27ae60,color:white
    classDef quantization fill:#9b59b6,stroke:#8e44ad,color:white
    classDef model fill:#e67e22,stroke:#d35400,color:white
    classDef input fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef output fill:#95a5a6,stroke:#7f8c8d,color:white
```
