# vllm-project/vllm-neuron architecture

> Architecture diagram of the GitHub repository vllm-project/vllm-neuron, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vllm-project/vllm-neuron
- Repository: https://github.com/vllm-project/vllm-neuron
- Video tour (about a minute): https://gitdiagram.com/vllm-project/vllm-neuron/video

## Overview

vLLM Neuron is a beta vLLM platform plugin that runs supported language, embedding, and multimodal models on AWS Trainium. A request enters through vLLM’s compatible serving APIs, is scheduled and executed by Neuron workers, and produces generated tokens or embeddings. The repository also contains distributed execution, disaggregated encoder routing, speculative decoding, accuracy-debugging and snapshot capture, and observability components. The README documents these capabilities; sampled source excerpts are partial, so uncertain internal wiring is omitted.

Main entry points: [`vllm_neuron/vllm/worker/neuron_model_runner.py`](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_model_runner.py) (Model runner), [`vllm_neuron/vllm/worker/neuron_worker.py`](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_worker.py) (Neuron worker), [`vllm_neuron/vllm/disaggregated_encoder/router.py`](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/disaggregated_encoder/router.py) (Encoder router), [`vllm_neuron/vllm/platform.py`](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/platform.py) (Neuron platform)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving and execution"]
  node_platform["Neuron platform<br/>[platform.py]"]
  node_scheduler["Request scheduler<br/>[scheduler.py]"]
  node_worker["Neuron worker<br/>[neuron_worker.py]"]
  node_runner["Model runner"]
  node_specdecode["EAGLE decoding<br/>[eagle.py]"]
end

subgraph group_compute["Models and compute"]
  node_registry["Model registry<br/>[registry.py]"]
  node_models["Model implementations"]
  node_ops["Neuron operations"]
  node_sampling["Sampling<br/>[sampler.py]"]
  node_kv["KV cache<br/>[kv_cache.py]"]
end

subgraph group_distributed["Distributed inference"]
  node_parallel["Parallel state"]
  node_collectives["Collectives<br/>[all2all.py]"]
  node_epd["Encoder router<br/>[router.py]"]
end

subgraph group_operations["Debug and observability"]
  node_accuracy["Accuracy debugger<br/>[api.py]"]
  node_validation["Logit validation"]
  node_snapshot["Tensor snapshots<br/>[capture.py]"]
  node_metrics["Serving metrics<br/>[metrics.py]"]
  node_profiler["Neuron profiler<br/>[neuron_profiler.py]"]
end

node_client(("API client"))
node_vllm["vLLM serving APIs"]
node_ve["Vision encoders"]
node_pd["Prefill/decode servers"]
node_trainium["AWS Trainium"]

node_client -->|"sends requests"| node_vllm
node_vllm -->|"selects backend"| node_platform
node_platform -->|"integrates scheduling"| node_scheduler
node_scheduler -->|"dispatches work"| node_worker
node_worker -->|"loads and runs"| node_runner
node_runner -->|"resolves model"| node_registry
node_registry -->|"selects implementation"| node_models
node_models -->|"uses operations"| node_ops
node_runner -->|"samples outputs"| node_sampling
node_runner -->|"binds cache"| node_kv
node_runner -->|"uses groups"| node_parallel
node_parallel -->|"coordinates ranks"| node_collectives
node_runner -->|"executes on"| node_trainium
node_epd -.->|"routes encoding"| node_ve
node_epd -.->|"forwards requests"| node_pd
node_runner -.->|"supports decoding"| node_specdecode
node_accuracy -->|"runs validation"| node_validation
node_runner -.->|"publishes context"| node_snapshot
node_worker -->|"creates profiler"| node_profiler
node_runner -.->|"reports metrics"| node_metrics

click node_platform "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/platform.py"
click node_scheduler "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/core/scheduler.py"
click node_worker "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_worker.py"
click node_runner "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_model_runner.py"
click node_registry "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/model/registry.py"
click node_models "https://github.com/vllm-project/vllm-neuron/tree/release-0.24.0.1.1.0/vllm_neuron/model"
click node_ops "https://github.com/vllm-project/vllm-neuron/tree/release-0.24.0.1.1.0/vllm_neuron/functional"
click node_sampling "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/nn/sampler.py"
click node_kv "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/model/kv_cache.py"
click node_parallel "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/parallel/neuron_parallel_state.py"
click node_collectives "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/parallel/all2all.py"
click node_epd "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/disaggregated_encoder/router.py"
click node_specdecode "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/spec_decode/eagle.py"
click node_accuracy "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/accuracy/accuracy_debugger/api.py"
click node_validation "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/accuracy/logit_validation.py"
click node_snapshot "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/snapshot/capture.py"
click node_metrics "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/metrics.py"
click node_profiler "https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_profiler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_platform,node_scheduler,node_worker,node_runner,node_specdecode,node_client toneBlue
class node_registry,node_models,node_ops,node_sampling,node_kv toneAmber
class node_parallel,node_collectives,node_epd,node_vllm,node_pd toneMint
class node_accuracy,node_validation,node_snapshot,node_metrics,node_profiler toneRose
class node_ve,node_trainium toneIndigo
```

## Components

### Serving and execution

- [Neuron platform](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/platform.py): `vllm_neuron/vllm/platform.py`
- [Request scheduler](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/core/scheduler.py): `vllm_neuron/vllm/core/scheduler.py`
- [Neuron worker](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_worker.py): `vllm_neuron/vllm/worker/neuron_worker.py`
- [Model runner](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_model_runner.py): `vllm_neuron/vllm/worker/neuron_model_runner.py`
- [EAGLE decoding](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/spec_decode/eagle.py): `vllm_neuron/vllm/spec_decode/eagle.py`

### Models and compute

- [Model registry](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/model/registry.py): `vllm_neuron/model/registry.py`
- [Model implementations](https://github.com/vllm-project/vllm-neuron/tree/release-0.24.0.1.1.0/vllm_neuron/model): `vllm_neuron/model`
- [Neuron operations](https://github.com/vllm-project/vllm-neuron/tree/release-0.24.0.1.1.0/vllm_neuron/functional): `vllm_neuron/functional`
- [Sampling](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/nn/sampler.py): `vllm_neuron/nn/sampler.py`
- [KV cache](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/model/kv_cache.py): `vllm_neuron/model/kv_cache.py`

### Distributed inference

- [Parallel state](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/parallel/neuron_parallel_state.py): `vllm_neuron/parallel/neuron_parallel_state.py`
- [Collectives](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/parallel/all2all.py): `vllm_neuron/parallel/all2all.py`
- [Encoder router](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/disaggregated_encoder/router.py): `vllm_neuron/vllm/disaggregated_encoder/router.py`

### Debug and observability

- [Accuracy debugger](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/accuracy/accuracy_debugger/api.py): `vllm_neuron/accuracy/accuracy_debugger/api.py`
- [Logit validation](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/accuracy/logit_validation.py): `vllm_neuron/accuracy/logit_validation.py`
- [Tensor snapshots](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/snapshot/capture.py): `vllm_neuron/snapshot/capture.py`
- [Serving metrics](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/metrics.py): `vllm_neuron/metrics.py`
- [Neuron profiler](https://github.com/vllm-project/vllm-neuron/blob/release-0.24.0.1.1.0/vllm_neuron/vllm/worker/neuron_profiler.py): `vllm_neuron/vllm/worker/neuron_profiler.py`

### Other components

- **API client**
- **vLLM serving APIs**
- **Vision encoders**
- **Prefill/decode servers**
- **AWS Trainium**

## Connections

- API client → vLLM serving APIs: sends requests
- vLLM serving APIs → Neuron platform: selects backend
- Neuron platform → Request scheduler: integrates scheduling
- Request scheduler → Neuron worker: dispatches work
- Neuron worker → Model runner: loads and runs
- Model runner → Model registry: resolves model
- Model registry → Model implementations: selects implementation
- Model implementations → Neuron operations: uses operations
- Model runner → Sampling: samples outputs
- Model runner → KV cache: binds cache
- Model runner → Parallel state: uses groups
- Parallel state → Collectives: coordinates ranks
- Model runner → AWS Trainium: executes on
- Encoder router → Vision encoders: routes encoding
- Encoder router → Prefill/decode servers: forwards requests
- Model runner → EAGLE decoding: supports decoding
- Accuracy debugger → Logit validation: runs validation
- Model runner → Tensor snapshots: publishes context
- Neuron worker → Neuron profiler: creates profiler
- Model runner → Serving metrics: reports metrics