# zhongw117/arcticinference architecture

> Architecture diagram of the GitHub repository zhongw117/arcticinference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhongw117/arcticinference
- Repository: https://github.com/zhongw117/arcticinference

## Overview

Arctic Inference is a vLLM plugin and inference-optimization library for LLM generation and embeddings. The primary workflow is a user request through familiar vLLM APIs or the optional multi-model server, into model execution and optimized decoding, then back as generated output. Separate branches cover embedding serving, reasoning evaluation, and semi-persistent model orchestration. The README documents several capabilities whose internal wiring was not sampled; those components are included without guessed call paths. Evidence is limited to the README and the supplied source excerpts and index.

Main entry points: [`arctic_inference/vllm/model_runner.py`](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/model_runner.py) (Model Execution), [`arctic_inference/semi_persistence/orchestrator.py`](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/orchestrator.py) (Model Orchestrator), [`arctic_inference/vllm/plugin.py`](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/plugin.py) (vLLM Plugin), [`arctic_inference/server/worker.py`](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/worker.py) (vLLM Worker)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving"]
  node_vllm_api["vLLM APIs"]
  node_server_api["Server API<br/>[api.py]"]
  node_server_pipeline["Dataset Pipeline<br/>[pipeline.py]"]
  node_replica_pool["Replica Pool<br/>[replica_pool.py]"]
  node_server_worker["vLLM Worker<br/>[worker.py]"]
  node_weight_sync["Weight Sync<br/>[engine.py]"]
end

subgraph group_runtime["vLLM Runtime"]
  node_plugin["vLLM Plugin<br/>[plugin.py]"]
  node_patching["Runtime Patching<br/>[patching.py]"]
  node_config["Arctic Config<br/>[config.py]"]
  node_model_runner["Model Execution<br/>[model_runner.py]"]
  node_sampling["Sampling<br/>[sampling.py]"]
end

subgraph group_optimization["Inference Optimizations"]
  node_speculator["Arctic Speculator"]
  node_proposer["Arctic Proposer<br/>[arctic_proposer.py]"]
  node_suffix[("Suffix Decoding<br/>[cache.py]")]
  node_ulysses["Ulysses Parallelism<br/>[ulysses.py]"]
  node_swiftkv["SwiftKV<br/>[llama_swiftkv.py]"]
end

subgraph group_embedding_reasoning["Embeddings &amp; Reasoning"]
  node_embedding_client["Embedding Client<br/>[client.py]"]
  node_embedding_manager["Embedding Replicas<br/>[replica_manager.py]"]
  node_embedding_replica["Embedding Replica<br/>[replica.py]"]
  node_dynasor["Reasoning Evaluation<br/>[evaluator.py]"]
end

subgraph group_semip["Semi-Persistence"]
  node_semi_client["Semi-Persistence Client<br/>[client.py]"]
  node_orchestrator["Model Orchestrator<br/>[orchestrator.py]"]
  node_model_instance["Model Instance<br/>[instance.py]"]
  node_semi_worker["Process Worker<br/>[worker.py]"]
  node_vllm_child["vLLM Child<br/>[vllm_child.py]"]
  node_slots[("GPU Slots<br/>[slots.py]")]
  node_op_pipeline["Operation Pipeline<br/>[pipeline.py]"]
end

node_user(("Inference User"))

node_user -->|"submits request"| node_vllm_api
node_vllm_api -->|"loads plugin"| node_plugin
node_plugin -.->|"applies patches"| node_patching
node_plugin -.->|"registers config"| node_config
node_config -->|"configures execution"| node_model_runner
node_model_runner -->|"uses speculation"| node_speculator
node_model_runner -->|"requests proposals"| node_proposer
node_model_runner -->|"uses suffix decoding"| node_suffix
node_model_runner -.->|"samples tokens"| node_sampling
node_user -.->|"requests serving"| node_server_api
node_server_api -.->|"dispatches generation"| node_replica_pool
node_server_pipeline -->|"submits generation"| node_replica_pool
node_replica_pool -->|"manages workers"| node_server_worker
node_server_worker -->|"configures extension"| node_weight_sync
node_server_worker -->|"loads plugins"| node_plugin
node_model_runner -.->|"uses parallelism"| node_ulysses
node_model_runner -.->|"runs optimized model"| node_swiftkv
node_user -.->|"requests embeddings"| node_embedding_client
node_embedding_client -.->|"forwards RPC"| node_embedding_manager
node_embedding_manager -.->|"routes requests"| node_embedding_replica
node_user -.->|"evaluates reasoning"| node_dynasor
node_semi_client -->|"sends operations"| node_orchestrator
node_orchestrator -->|"schedules operations"| node_op_pipeline
node_orchestrator -->|"manages instances"| node_model_instance
node_orchestrator -->|"allocates GPU slots"| node_slots
node_model_instance -->|"starts worker"| node_semi_worker
node_semi_worker -->|"spawns child"| node_vllm_child

click node_plugin "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/plugin.py"
click node_patching "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/patching.py"
click node_config "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/config.py"
click node_model_runner "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/model_runner.py"
click node_sampling "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/sampling.py"
click node_speculator "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_speculator.py"
click node_proposer "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_proposer.py"
click node_suffix "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/suffix_decoding/cache.py"
click node_ulysses "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/ulysses.py"
click node_swiftkv "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/swiftkv/llama_swiftkv.py"
click node_server_api "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/api.py"
click node_server_pipeline "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/pipeline.py"
click node_replica_pool "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/replica_pool.py"
click node_server_worker "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/worker.py"
click node_weight_sync "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/weight_sync/engine.py"
click node_embedding_client "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/client.py"
click node_embedding_manager "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/replica_manager.py"
click node_embedding_replica "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/replica.py"
click node_dynasor "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/dynasor/evaluator.py"
click node_semi_client "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/client.py"
click node_orchestrator "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/orchestrator.py"
click node_model_instance "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/instance.py"
click node_semi_worker "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/worker.py"
click node_vllm_child "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/vllm_child.py"
click node_slots "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/slots.py"
click node_op_pipeline "https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vllm_api,node_server_api,node_server_pipeline,node_replica_pool,node_server_worker,node_weight_sync,node_user toneBlue
class node_plugin,node_patching,node_config,node_model_runner,node_sampling toneAmber
class node_speculator,node_proposer,node_suffix,node_ulysses,node_swiftkv toneMint
class node_embedding_client,node_embedding_manager,node_embedding_replica,node_dynasor toneRose
class node_semi_client,node_orchestrator,node_model_instance,node_semi_worker,node_vllm_child,node_slots,node_op_pipeline toneIndigo
```

## Components

### Serving

- **vLLM APIs**
- [Server API](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/api.py): `arctic_inference/server/api.py`
- [Dataset Pipeline](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/pipeline.py): `arctic_inference/server/pipeline.py`
- [Replica Pool](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/replica_pool.py): `arctic_inference/server/replica_pool.py`
- [vLLM Worker](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/worker.py): `arctic_inference/server/worker.py`
- [Weight Sync](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/server/weight_sync/engine.py): `arctic_inference/server/weight_sync/engine.py`

### vLLM Runtime

- [vLLM Plugin](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/plugin.py): `arctic_inference/vllm/plugin.py`
- [Runtime Patching](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/patching.py): `arctic_inference/patching.py`
- [Arctic Config](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/config.py): `arctic_inference/vllm/config.py`
- [Model Execution](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/model_runner.py): `arctic_inference/vllm/model_runner.py`
- [Sampling](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/sampling.py): `arctic_inference/vllm/sampling.py`

### Inference Optimizations

- [Arctic Speculator](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_speculator.py): `arctic_inference/vllm/spec_dec/arctic_speculator.py`
- [Arctic Proposer](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/spec_dec/arctic_proposer.py): `arctic_inference/vllm/spec_dec/arctic_proposer.py`
- [Suffix Decoding](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/suffix_decoding/cache.py): `arctic_inference/suffix_decoding/cache.py`
- [Ulysses Parallelism](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/ulysses.py): `arctic_inference/vllm/ulysses.py`
- [SwiftKV](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/vllm/swiftkv/llama_swiftkv.py): `arctic_inference/vllm/swiftkv/llama_swiftkv.py`

### Embeddings & Reasoning

- [Embedding Client](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/client.py): `arctic_inference/embedding/client.py`
- [Embedding Replicas](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/replica_manager.py): `arctic_inference/embedding/replica_manager.py`
- [Embedding Replica](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/embedding/replica.py): `arctic_inference/embedding/replica.py`
- [Reasoning Evaluation](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/dynasor/evaluator.py): `arctic_inference/dynasor/evaluator.py`

### Semi-Persistence

- [Semi-Persistence Client](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/client.py): `arctic_inference/semi_persistence/client.py`
- [Model Orchestrator](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/orchestrator.py): `arctic_inference/semi_persistence/orchestrator.py`
- [Model Instance](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/instance.py): `arctic_inference/semi_persistence/instance.py`
- [Process Worker](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/worker.py): `arctic_inference/semi_persistence/worker.py`
- [vLLM Child](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/vllm_child.py): `arctic_inference/semi_persistence/vllm_child.py`
- [GPU Slots](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/slots.py): `arctic_inference/semi_persistence/slots.py`
- [Operation Pipeline](https://github.com/zhongw117/arcticinference/blob/main/arctic_inference/semi_persistence/pipeline.py): `arctic_inference/semi_persistence/pipeline.py`

### Other components

- **Inference User**

## Connections

- Inference User → vLLM APIs: submits request
- vLLM APIs → vLLM Plugin: loads plugin
- vLLM Plugin → Runtime Patching: applies patches
- vLLM Plugin → Arctic Config: registers config
- Arctic Config → Model Execution: configures execution (evidence: `arctic_inference/vllm/model_runner.py`)
- Model Execution → Arctic Speculator: uses speculation (evidence: `arctic_inference/vllm/model_runner.py`)
- Model Execution → Arctic Proposer: requests proposals (evidence: `arctic_inference/vllm/model_runner.py`)
- Model Execution → Suffix Decoding: uses suffix decoding (evidence: `arctic_inference/vllm/model_runner.py`)
- Model Execution → Sampling: samples tokens
- Inference User → Server API: requests serving
- Server API → Replica Pool: dispatches generation
- Dataset Pipeline → Replica Pool: submits generation (evidence: `arctic_inference/server/pipeline.py`)
- Replica Pool → vLLM Worker: manages workers (evidence: `arctic_inference/server/replica_pool.py`)
- vLLM Worker → Weight Sync: configures extension (evidence: `arctic_inference/server/worker.py`)
- vLLM Worker → vLLM Plugin: loads plugins (evidence: `arctic_inference/server/worker.py`)
- Model Execution → Ulysses Parallelism: uses parallelism
- Model Execution → SwiftKV: runs optimized model
- Inference User → Embedding Client: requests embeddings
- Embedding Client → Embedding Replicas: forwards RPC
- Embedding Replicas → Embedding Replica: routes requests
- Inference User → Reasoning Evaluation: evaluates reasoning
- Semi-Persistence Client → Model Orchestrator: sends operations (evidence: `arctic_inference/semi_persistence/client.py`)
- Model Orchestrator → Operation Pipeline: schedules operations (evidence: `arctic_inference/semi_persistence/orchestrator.py`)
- Model Orchestrator → Model Instance: manages instances (evidence: `arctic_inference/semi_persistence/orchestrator.py`)
- Model Orchestrator → GPU Slots: allocates GPU slots (evidence: `arctic_inference/semi_persistence/orchestrator.py`)
- Model Instance → Process Worker: starts worker (evidence: `arctic_inference/semi_persistence/instance.py`)
- Process Worker → vLLM Child: spawns child (evidence: `arctic_inference/semi_persistence/worker.py`)
