How zhongw117/arcticinference works
Arctic Inference is a vLLM plugin and inference-optimization library for LLM generation and embeddings. The primary workflow is a user request through familiar vLLM APIs or the optional multi-model server, into model execution and optimized decoding, then back as generated output. Separate branches cover embedding serving, reasoning evaluation, and semi-persistent model orchestration. The README documents several capabilities whose internal wiring was not sampled; those components are included without guessed call paths. Evidence is limited to the README and the supplied source excerpts and index.
Main entry points: model_runner.py Model Execution · orchestrator.py Model Orchestrator · plugin.py vLLM Plugin · worker.py vLLM Worker
Components 28
Serving
- vLLM APIs
- Server API
arctic_inference/server/api.py - Dataset Pipeline
arctic_inference/server/pipeline.py - Replica Pool
arctic_inference/server/replica_pool.py - vLLM Worker
arctic_inference/server/worker.py - Weight Sync
arctic_inference/server/weight_sync/engine.py
vLLM Runtime
- vLLM Plugin
arctic_inference/vllm/plugin.py - Runtime Patching
arctic_inference/patching.py - Arctic Config
arctic_inference/vllm/config.py - Model Execution
arctic_inference/vllm/model_runner.py - Sampling
arctic_inference/vllm/sampling.py
Inference Optimizations
- Arctic Speculator
arctic_inference/vllm/spec_dec/arctic_speculator.py - Arctic Proposer
arctic_inference/vllm/spec_dec/arctic_proposer.py - Suffix Decoding
arctic_inference/suffix_decoding/cache.py - Ulysses Parallelism
arctic_inference/vllm/ulysses.py - SwiftKV
arctic_inference/vllm/swiftkv/llama_swiftkv.py
Embeddings & Reasoning
- Embedding Client
arctic_inference/embedding/client.py - Embedding Replicas
arctic_inference/embedding/replica_manager.py - Embedding Replica
arctic_inference/embedding/replica.py - Reasoning Evaluation
arctic_inference/dynasor/evaluator.py
Semi-Persistence
- Semi-Persistence Client
arctic_inference/semi_persistence/client.py - Model Orchestrator
arctic_inference/semi_persistence/orchestrator.py - Model Instance
arctic_inference/semi_persistence/instance.py - Process Worker
arctic_inference/semi_persistence/worker.py - vLLM Child
arctic_inference/semi_persistence/vllm_child.py - GPU Slots
arctic_inference/semi_persistence/slots.py - Operation Pipeline
arctic_inference/semi_persistence/pipeline.py
Other components
- Inference User
Connections 27
- Inference User → vLLM APIs: submits request
- vLLM APIs → vLLM Plugin: loads plugin
- vLLM Plugin → Runtime Patching: applies patches
- vLLM Plugin → Arctic Config: registers config
- Arctic Config → Model Execution: configures execution
arctic_inference/vllm/model_runner.py - Model Execution → Arctic Speculator: uses speculation
arctic_inference/vllm/model_runner.py - Model Execution → Arctic Proposer: requests proposals
arctic_inference/vllm/model_runner.py - Model Execution → Suffix Decoding: uses suffix decoding
arctic_inference/vllm/model_runner.py - Model Execution → Sampling: samples tokens
- Inference User → Server API: requests serving
- Server API → Replica Pool: dispatches generation
- Dataset Pipeline → Replica Pool: submits generation
arctic_inference/server/pipeline.py - Replica Pool → vLLM Worker: manages workers
arctic_inference/server/replica_pool.py - vLLM Worker → Weight Sync: configures extension
arctic_inference/server/worker.py - vLLM Worker → vLLM Plugin: loads plugins
arctic_inference/server/worker.py - Model Execution → Ulysses Parallelism: uses parallelism
- Model Execution → SwiftKV: runs optimized model
- Inference User → Embedding Client: requests embeddings
- Embedding Client → Embedding Replicas: forwards RPC
- Embedding Replicas → Embedding Replica: routes requests
- Inference User → Reasoning Evaluation: evaluates reasoning
- Semi-Persistence Client → Model Orchestrator: sends operations
arctic_inference/semi_persistence/client.py - Model Orchestrator → Operation Pipeline: schedules operations
arctic_inference/semi_persistence/orchestrator.py - Model Orchestrator → Model Instance: manages instances
arctic_inference/semi_persistence/orchestrator.py - Model Orchestrator → GPU Slots: allocates GPU slots
arctic_inference/semi_persistence/orchestrator.py - Model Instance → Process Worker: starts worker
arctic_inference/semi_persistence/instance.py - Process Worker → vLLM Child: spawns child
arctic_inference/semi_persistence/worker.py