# turboderp-org/exllamav2 architecture

> Architecture diagram of the GitHub repository turboderp-org/exllamav2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/turboderp-org/exllamav2
- Repository: https://github.com/turboderp-org/exllamav2

## Overview

##### Project type and purpose
- ExLlamaV2 is an open-source inference library for running large language models (LLMs) locally on consumer GPUs.
- It exposes a Python API, backs it with a Torch C++/CUDA extension (“exllamav2_ext”), and provides scripts for model conversion, evaluation, and example applications (chat, bulk inference, websocket server).

##### File structure
- Top level:
- Python entry points and utilities: `convert.py`, `model_diff.py`, `test_inference.py`
- Main package: `exllamav2/` containing pure-Python modules (architecture, attention, tokenizer, generator, server, util)
- Native extension: `exllamav2/exllamav2_ext/` with C++ and CUDA kernels for compute-intensive operations
- Conversion helpers: `exllamav2/conversion/` and top-level `util/`
- Examples: `examples/` folder demonstrating usage scenarios (chat, bulk inference, async, websocket)
- Tests: `tests/` and `eval/` for unit tests and evaluation scripts
- Configuration: `setup.py`, `requirements.txt`, GitHub workflows

##### README for additional insights
- Key features: dynamic batching generator, quantization (“EXL2”), FlashAttention support, integration points (TabbyAPI, ExUI, text-generation-webui).
- Usage patterns: Python API → generator → model → extension → GPU.

##### Main components
a. Python API layer
- Modules: `exllamav2/config.py`, `exllamav2/tokenizer/`, `exllamav2/architecture.py`, `exllamav2/model.py`, `exllamav2/generator/`, `exllamav2/server/`
- Users interact via scripts in `examples/` or third-party servers (TabbyAPI, ExUI) that call the Python API.
b. Native extension layer
- Torch C++ extension (`exllamav2_ext`): high-performance operators (attention, quantize, sampling, kernels in C++/CUDA).
c. Conversion & quantization subsystem
- Scripts and modules under `exllamav2/conversion/` + `util/convert_safetensors.py`, `util/shard.py`, etc.
- Produces optimized, quantized model files for inference.
d. Generator & caching
- `exllamav2/generator/dynamic.py` (dynamic batching, KV cache deduplication) and filters/hooks.
e. Tokenizer
- `exllamav2/tokenizer` (HuggingFace or SentencePiece backends).
f. External services
- TabbyAPI or custom websocket server (`exllamav2/server/websocket.py`) that front-ends the library over REST/WebSocket.

##### Relationships & data flow
- User code or external API → Python API (config, tokenizer, model_init)
- Tokenizer encodes text → input IDs
- Model module dispatches forward calls to C++ extension kernels on GPU
- Generator orchestrates multi-step sampling using extension’s attention/MLP layers, maintaining KV cache
- Output tokens passed back to Python → decoded via tokenizer → returned to user
- Conversion tools (offline) transform raw model weights → optimized quantized artifacts consumed by model_init

##### Architectural patterns & design principles
- Layered architecture separating:
1. Interface layer (Python API, server)
2. Core logic layer (model definitions, generator)
3. Performance layer (C++/CUDA extension)
4. Utility layer (conversion, evaluation scripts, examples)
- Plugin-style extensibility: new quantization schemes, generators, filters, server back-ends
- Data-flow emphasis: clear pipeline from text → tokens → tensors → GPU kernels → tokens → text

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interface Layer
    subgraph "Interface Layer"
        direction TB
        ExamplesChat["examples/chat.py"]:::utility
        ExamplesBulk["examples/bulk_inference.py"]:::utility
        ExamplesAsync["examples/inference_async.py"]:::utility
        ExamplesWS["examples/ws_server.py"]:::utility
        ExternalTabby["TabbyAPI"]:::external
        ExternalExUI["ExUI"]:::external
    end

    %% Python API Layer
    subgraph "Python API Layer"
        direction TB
        Config["exllamav2/config.py"]:::python
        Architecture["exllamav2/architecture.py"]:::python
        Model["exllamav2/model.py"]:::python
        ModelInit["exllamav2/model_init.py"]:::python
        CompatUtil["exllamav2/compat.py"]:::python
        Util["exllamav2/util.py"]:::python
    end

    %% Tokenizer Subsystem
    subgraph "Tokenizer Subsystem"
        direction TB
        TokenizerMain["exllamav2/tokenizer/tokenizer.py"]:::python
        TokenizerBase["exllamav2/tokenizer/base.py"]:::python
        TokenizerHF["exllamav2/tokenizer/hf.py"]:::python
        TokenizerSPM["exllamav2/tokenizer/spm.py"]:::python
    end

    %% Generator & Caching Subsystem
    subgraph "Generator Subsystem"
        direction TB
        GeneratorBase["exllamav2/generator/base.py"]:::python
        GeneratorDynamic["exllamav2/generator/dynamic.py"]:::python
        GeneratorAsync["exllamav2/generator/dynamic_async.py"]:::python
        GeneratorFilters["exllamav2/generator/filters/"]:::python
        GeneratorHooks["exllamav2/generator/hooks.py"]:::python
    end

    %% Server Layer
    subgraph "Server Layer"
        direction TB
        ServerWS["exllamav2/server/websocket.py"]:::python
        ServerWSActions["exllamav2/server/websocket_actions.py"]:::python
    end

    %% Native Extension Layer
    subgraph "Native Extension" 
        direction TB
        Extension["exllamav2/exllamav2_ext/ (cpp/, cuda/)"]:::native
    end

    %% Conversion & Quantization
    subgraph "Conversion Subsystem"
        direction TB
        ConversionScripts["exllamav2/conversion/*.py"]:::utility
        UtilHelpers["util/convert_safetensors.py\nutil/shard.py\nutil/unshard.py\nconvert.py"]:::utility
        ModelArtifacts["Quantized Model Artifacts"]:::artifact
    end

    %% Evaluation & Testing
    subgraph "Evaluation & Testing"
        direction TB
        EvalScripts["eval/*.py"]:::utility
        Tests["tests/*.py"]:::utility
    end

    %% Data Flow
    ExamplesChat -->|uses API| Config
    ExamplesBulk -->|uses API| Config
    ExamplesAsync -->|uses API| Config
    ExamplesWS -->|uses API| Config

    ExternalTabby -->|calls| ServerWS
    ExternalExUI -->|calls| ServerWS

    ServerWS -->|invokes| Config
    ServerWSActions -->|invokes| Model

    Config -->|provides| ModelInit
    Architecture -->|defines| Model
    ModelInit -->|initializes| Model
    CompatUtil -->|utilities| Model
    Util -->|utilities| GeneratorDynamic

    TokenizerMain -->|encodes/decodes| TokenizerBase
    TokenizerMain -->|backend| TokenizerHF
    TokenizerMain -->|backend| TokenizerSPM
    Config -->|passes params to| TokenizerMain

    Model -->|dispatches forward| Extension
    TokenizerMain -->|provides IDs| Model

    GeneratorBase -->|orchestrates| Extension
    GeneratorDynamic -->|implements batching| GeneratorBase
    GeneratorAsync -->|async support| GeneratorBase
    GeneratorFilters -->|apply filters| GeneratorDynamic
    GeneratorHooks -->|apply hooks| GeneratorDynamic

    Model -->|uses| GeneratorBase
    Extension -->|executes kernels| GeneratorDynamic

    ConversionScripts -.->|produces| ModelArtifacts
    UtilHelpers -.->|supports| ConversionScripts
    ModelInit -.->|loads| ModelArtifacts

    ModelArtifacts -->|consumed by| ModelInit

    EvalScripts -->|tests| Model
    Tests -->|unit tests| TokenizerBase

    %% Click Events
    click Config "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/config.py"
    click Architecture "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/architecture.py"
    click Model "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/model.py"
    click ModelInit "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/model_init.py"
    click CompatUtil "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/compat.py"
    click Util "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/util.py"
    click TokenizerMain "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/tokenizer/tokenizer.py"
    click TokenizerBase "https://github.com/turboderp-org/exllamav2/blob/master/exllamavav2/tokenizer/base.py"
    click TokenizerHF "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/tokenizer/hf.py"
    click TokenizerSPM "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/tokenizer/spm.py"
    click GeneratorBase "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/generator/base.py"
    click GeneratorDynamic "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/generator/dynamic.py"
    click GeneratorAsync "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/generator/dynamic_async.py"
    click GeneratorFilters "https://github.com/turboderp-org/exllamav2/tree/master/exllamav2/generator/filters/"
    click GeneratorHooks "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/generator/hooks.py"
    click ServerWS "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/server/websocket.py"
    click ServerWSActions "https://github.com/turboderp-org/exllamav2/blob/master/exllamav2/server/websocket_actions.py"
    click Extension "https://github.com/turboderp-org/exllamav2/tree/master/exllamav2/exllamav2_ext/"
    click ConversionScripts "https://github.com/turboderp-org/exllamav2/tree/master/exllamav2/conversion/"
    click UtilHelpers "https://github.com/turboderp-org/exllamav2/blob/master/util/convert_safetensors.py"
    click ExamplesChat "https://github.com/turboderp-org/exllamav2/blob/master/examples/chat.py"
    click ExamplesBulk "https://github.com/turboderp-org/exllamav2/blob/master/examples/bulk_inference.py"
    click ExamplesAsync "https://github.com/turboderp-org/exllamav2/blob/master/examples/inference_async.py"
    click ExamplesWS "https://github.com/turboderp-org/exllamav2/blob/master/examples/ws_server.py"

    %% Styles
    classDef python fill:#e0f7fa,stroke:#0288d1,stroke-width:1px
    classDef native fill:#90caf9,stroke:#1565c0,stroke-width:1px
    classDef utility fill:#fff9c4,stroke:#f9a825,stroke-width:1px
    classDef external fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    classDef artifact fill:#eceff1,stroke:#546e7a,stroke-width:1px,shape:cylinder
```
