# snowflakedb/arcticinference architecture

> Architecture diagram of the GitHub repository snowflakedb/arcticinference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snowflakedb/arcticinference
- Repository: https://github.com/snowflakedb/arcticinference

## Overview

##### Project type and purpose
- ArcticInference is an open-source Python library (with a small C++ component) that provides inference optimizations for large language models by plugging into vLLM v0.8.1.
- Its purpose is to automatically patch vLLM to use Snowflake AI Research’s optimizations—Arctic Ulysses, SwiftKV, Arctic Speculator, etc.—while exposing the same vLLM API/CLI to users.

##### File structure
- Top-level modules
- arctic_inference/common: shared utilities, suffix_cache simulator and production code, SwiftKV config
- arctic_inference/vllm: plugin entry points (args, config), model runner, plugin hooks, and code for each optimization
- csrc/suffix_cache: C++ implementation of suffix cache, with pybind.cc binding to Python
- projects/: example applications for each optimization (offline inference scripts plus project-specific README)
- tests/: unit tests (e.g., test_patching.py)
- scripts/: license check and release automation
- setup.py, pyproject.toml: packaging and build configuration
- This structure reveals a layered architecture:
1. Core utilities (common)
2. vLLM plugin layer (vllm)
3. Native extension (C++)
4. Example applications

##### README for additional insights
- Confirms integration with vLLM custom plugin interface
- Lists three main “projects” (SwiftKV, Ulysses, Speculator) as pluggable inference optimizations

##### Main components
a. ArcticInference Python library
- Common utilities (suffix_cache, SwiftKV configs)
- vLLM plugin module (args, config, model_runner, plugin hooks, spec_dec, plugins for each optimization)
b. C++ suffix_cache native extension
- suffix_tree.cc/.h, pybind.cc
c. vLLM inference engine (external dependency)
d. User interface
- Python API calls or CLI
e. Example applications under projects/
f. Build & packaging
- setup.py, pyproject.toml, scripts/

##### Relationships and data flow
- User issues an inference request via the ArcticInference-patched vLLM CLI or API
- vLLM core loads ArcticInference plugin
- Plugin code in arctic_inference/vllm intercepts requests and routes through:
1. Arctic Speculator (spec_dec)
2. SwiftKV paths (swiftkv)
3. Ulysses sequence parallelism
- If suffix cache is needed, Python code calls into the C++ extension via pybind
- Optimized logits pass back into vLLM’s inference pipeline
- Results return to the user

##### Architectural patterns & design principles
- Plugin pattern: ArcticInference hooks into vLLM via a well-defined plugin API
- Layered modularity: clear separation between common utilities, plugin entry, project implementations, native code
- Extensibility: new optimizations can be dropped into arctic_inference/vllm/spec_dec or arctic_inference/common/swiftkv
- Native extension integration: performance-critical suffix cache in C++

##### Technologies, frameworks, libraries
- Python 3.8+ packaging (setup.py, pyproject.toml)
- Pybind11 (pybind.cc) for C++/Python bridging
- vLLM v0.8.1 custom plugin interface
- CMake for building suffix_cache
- pytest for unit tests
- shell scripts for release automation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    User["User\n(API/CLI)"]:::external

    %% External vLLM Engine
    subgraph "vLLM Engine (External)" 
        direction TB
        vLLM["vLLM v0.8.1"]:::external
    end

    %% ArcticInference Plugin
    subgraph "ArcticInference Plugin" 
        direction TB
        subgraph "Common Utilities" 
            direction TB
            SuffixSimulator["simulator.py"]:::python
            click SuffixSimulator "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/common/suffix_cache/simulator.py"
            SuffixCachePy["suffix_cache.py"]:::python
            click SuffixCachePy "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/common/suffix_cache/suffix_cache.py"
            SwiftKVConfig["configs.py"]:::python
            click SwiftKVConfig "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/common/swiftkv/configs.py"
        end
        subgraph "vLLM Plugin Layer"
            direction TB
            Args["args.py"]:::python
            click Args "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/args.py"
            Config["config.py"]:::python
            click Config "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/config.py"
            ModelRunner["model_runner.py"]:::python
            click ModelRunner "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/model_runner.py"
            PluginsEntry["plugins.py"]:::python
            click PluginsEntry "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/plugins.py"
            Patching["patching.py"]:::python
            click Patching "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/patching.py"
            subgraph "Speculator Module"
                direction TB
                SpecInit["__init__.py"]:::python
                click SpecInit "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/__init__.py"
                Proposer["arctic_proposer.py"]:::python
                click Proposer "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_proposer.py"
                Speculator["arctic_speculator.py"]:::python
                click Speculator "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_speculator.py"
                FP8["fp8.py"]:::python
                click FP8 "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/fp8.py"
                LogitsOpt["logits_processor_opt.py"]:::python
                click LogitsOpt "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/logits_processor_opt.py"
                VocabEmb["vocab_parallel_embedding.py"]:::python
                click VocabEmb "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_dec/vocab_parallel_embedding.py"
                SpecDec["spec_decoding.py"]:::python
                click SpecDec "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/spec_decoding.py"
            end
            SwiftKVPlugin["llama_swiftkv.py"]:::python
            click SwiftKVPlugin "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/swiftkv/llama_swiftkv.py"
            UlyssesPlugin["ulysses.py"]:::python
            click UlyssesPlugin "https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/vllm/ulysses.py"
        end
        subgraph "C++ Native Extension"
            direction TB
            Pybind["pybind.cc"]:::cpp
            click Pybind "https://github.com/snowflakedb/arcticinference/blob/main/csrc/suffix_cache/pybind.cc"
            SuffixTreeCC["suffix_tree.cc"]:::cpp
            click SuffixTreeCC "https://github.com/snowflakedb/arcticinference/blob/main/csrc/suffix_cache/suffix_tree.cc"
            SuffixTreeH["suffix_tree.h"]:::cpp
            click SuffixTreeH "https://github.com/snowflakedb/arcticinference/blob/main/csrc/suffix_cache/suffix_tree.h"
            CMake["CMakeLists.txt"]:::cpp
            click CMake "https://github.com/snowflakedb/arcticinference/blob/main/csrc/suffix_cache/CMakeLists.txt"
        end
    end

    %% Example Applications
    subgraph "Example Applications (Projects)"
        direction TB
        subgraph "Speculator Project"
            SpecApp["offline_inference_spec_dec.py"]:::python
            click SpecApp "https://github.com/snowflakedb/arcticinference/blob/main/projects/spec_dec/offline_inference_spec_dec.py"
            SpecReadme["README.md"]:::docs
            click SpecReadme "https://github.com/snowflakedb/arcticinference/blob/main/projects/spec_dec/README.md"
        end
        subgraph "SwiftKV Project"
            SwiftApp["offline_inference_swiftkv.py"]:::python
            click SwiftApp "https://github.com/snowflakedb/arcticinference/blob/main/projects/swiftkv/offline_inference_swiftkv.py"
            SwiftReadme["README.md"]:::docs
            click SwiftReadme "https://github.com/snowflakedb/arcticinference/blob/main/projects/swiftkv/README.md"
        end
        subgraph "Ulysses Project"
            UlyssesApp["offline_inference_ulysses.py"]:::python
            click UlyssesApp "https://github.com/snowflakedb/arcticinference/blob/main/projects/ulysses/offline_inference_ulysses.py"
            UlyssesReadme["README.md"]:::docs
            click UlyssesReadme "https://github.com/snowflakedb/arcticinference/blob/main/projects/ulysses/README.md"
        end
    end

    %% Build & Packaging
    subgraph "Build & Packaging"
        direction TB
        Setup["setup.py"]:::packaging
        click Setup "https://github.com/snowflakedb/arcticinference/blob/main/setup.py"
        Pyproject["pyproject.toml"]:::packaging
        click Pyproject "https://github.com/snowflakedb/arcticinference/blob/main/pyproject.toml"
        CheckLicense["check_license.py"]:::scripts
        click CheckLicense "https://github.com/snowflakedb/arcticinference/blob/main/scripts/check_license.py"
        ReleaseScript["release.sh"]:::scripts
        click ReleaseScript "https://github.com/snowflakedb/arcticinference/blob/main/scripts/release.sh"
    end

    %% Unit Tests
    Tests["test_patching.py"]:::tests
    click Tests "https://github.com/snowflakedb/arcticinference/blob/main/tests/test_patching.py"

    %% Connections
    User -->|"invoke API/CLI"| vLLM
    vLLM -->|"loads plugin"| Patching
    Patching -->|hook| Args
    Args --> Config
    Config --> ModelRunner
    ModelRunner --> PluginsEntry
    PluginsEntry --> SpecInit
    PluginsEntry --> SwiftKVPlugin
    PluginsEntry --> UlyssesPlugin
    Speculator --> ModelRunner
    SwiftKVPlugin -->|uses| SwiftKVConfig
    UlyssesPlugin -->|parallelizes| ModelRunner
    ModelRunner -->|calls| Pybind
    Pybind --> SuffixTreeCC
    SuffixTreeCC --> SuffixTreeH
    ModelRunner -->|returns logits| vLLM
    vLLM -->|"response"| User

    %% Examples usage
    User -.-> SpecApp
    User -.-> SwiftApp
    User -.-> UlyssesApp

    %% Packaging/Test flows
    Setup -->|build| Pybind
    Pyproject --> Setup
    CheckLicense --> Setup
    ReleaseScript --> Setup
    Tests -->|validates| Patching

    %% Styles
    classDef python fill:#AED6F1,stroke:#1B4F72,stroke-width:1px;
    classDef cpp fill:#FADBD8,stroke:#78281F,stroke-width:1px;
    classDef external fill:#D5D8DC,stroke:#283747,stroke-width:1px;
    classDef packaging fill:#FCF3CF,stroke:#7D6608,stroke-width:1px;
    classDef scripts fill:#FCF3CF,stroke:#7D6608,stroke-width:1px;
    classDef tests fill:#D6EAF8,stroke:#1B4F72,stroke-width:1px;
    classDef docs fill:#E8DAEF,stroke:#4A235A,stroke-width:1px;
```
