# 1314521gjy/ninfer-fusion-kvmem architecture

> Architecture diagram of the GitHub repository 1314521gjy/ninfer-fusion-kvmem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/1314521gjy/ninfer-fusion-kvmem
- Repository: https://github.com/1314521gjy/ninfer-fusion-kvmem

## Overview

NInfer is a CUDA inference engine and serving application, with downstream changes aimed at long-context generation on a smaller device KV pool. The principal workflow accepts a prompt, prepares it for generation, plans and runs model work, and reuses or retrieves context state as generation proceeds. The README documents content-ranked KVMem retrieval, host-backed checkpoint reuse, adaptive device profiles, speculative decoding, and multimodal support. Several subsystem links are documented at a high level; unsampled implementation details are not treated as verified call paths.

Main entry points: [`patches/changed-files/src/models/qwen3_5/program/program.h`](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/program.h) (Qwen program), [`patches/changed-files/src/runtime/engine/engine.cpp`](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/runtime/engine/engine.cpp) (Engine runtime), [`patches/changed-files/src/models/qwen3_5/program/storage/context.cpp`](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/storage/context.cpp) (Context storage), [`patches/changed-files/apps/serve/main.cpp`](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/apps/serve/main.cpp) (Serve entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving and API"]
  node_serve_main["Serve entry<br/>[main.cpp]"]
  node_generation_service["Generation service"]
end

subgraph group_generation["Prompt and Generation"]
  node_engine_api["Engine API<br/>[engine.h]"]
  node_engine_runtime["Engine runtime<br/>[engine.cpp]"]
  node_frontend["Prompt frontend"]
  node_program["Qwen program<br/>[program.h]"]
  node_prefill["Prompt prefill<br/>[prefill.cpp]"]
end

subgraph group_context["Context Management"]
  node_decoder_state["Decoder state<br/>[decoder_state.cpp]"]
  node_kv_cache[("Paged KV cache<br/>[paged_kv_cache.cpp]")]
  node_context_storage["Context storage<br/>[context.cpp]"]
  node_checkpoint_capture["Checkpoint capture<br/>[capture.cpp]"]
  node_checkpoint_materialize["Checkpoint materialization"]
  node_pressure_planning["Pressure planning<br/>[pressure.cpp]"]
  node_kvmem_retrieval["KVMem retrieval"]
end

subgraph group_model["Model Execution"]
  node_model_execution["Qwen execution"]
  node_speculative["Speculative decoding"]
  node_multimodal["Vision and media<br/>[vision.cpp]"]
end

subgraph group_platform["Runtime Platform"]
  node_device_runtime["CUDA device runtime<br/>[device.h]"]
  node_adaptive_profiles["Device profiles"]
end

node_user(("Prompt user"))

node_user -->|"submits prompt"| node_serve_main
node_serve_main -->|"starts service"| node_generation_service
node_generation_service -.->|"serves generation"| node_engine_api
node_engine_api -.->|"delegates execution"| node_engine_runtime
node_engine_runtime -.->|"prepares prompt"| node_frontend
node_engine_runtime -.->|"runs generation"| node_program
node_program -->|"prefills prompt"| node_prefill
node_program -.->|"uses state"| node_decoder_state
node_program -.->|"executes model"| node_model_execution
node_model_execution -.->|"uses CUDA"| node_device_runtime
node_program -.->|"supports speculation"| node_speculative
node_program -.->|"supports vision"| node_multimodal
node_context_storage -->|"manages KV"| node_kv_cache
node_checkpoint_capture -->|"captures checkpoints"| node_context_storage
node_pressure_planning -->|"plans materialization"| node_checkpoint_materialize
node_checkpoint_materialize -->|"materializes context"| node_context_storage
node_prefill -->|"retrieves relevant pages"| node_kvmem_retrieval
node_engine_runtime -.->|"selects device profile"| node_adaptive_profiles

click node_serve_main "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/apps/serve/main.cpp"
click node_generation_service "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/serve/generation_service.cpp"
click node_engine_api "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/include/ninfer/engine.h"
click node_engine_runtime "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/runtime/engine/engine.cpp"
click node_frontend "https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/frontend"
click node_program "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/program.h"
click node_prefill "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/prefill.cpp"
click node_decoder_state "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/state/decoder_state.cpp"
click node_kv_cache "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/core/paged_kv_cache.cpp"
click node_context_storage "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/storage/context.cpp"
click node_checkpoint_capture "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/transactions/capture.cpp"
click node_checkpoint_materialize "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/transactions/materialization.cpp"
click node_pressure_planning "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/planning/pressure.cpp"
click node_kvmem_retrieval "https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/ops"
click node_model_execution "https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/execution"
click node_speculative "https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/program/speculative"
click node_multimodal "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/models/qwen3_5/execution/vision.cpp"
click node_device_runtime "https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/core/device.h"
click node_adaptive_profiles "https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/runtime/engine"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_serve_main,node_generation_service,node_user toneBlue
class node_engine_api,node_engine_runtime,node_frontend,node_program,node_prefill toneAmber
class node_decoder_state,node_kv_cache,node_context_storage,node_checkpoint_capture,node_checkpoint_materialize,node_pressure_planning,node_kvmem_retrieval toneMint
class node_model_execution,node_speculative,node_multimodal toneRose
class node_device_runtime,node_adaptive_profiles toneIndigo
```

## Components

### Serving and API

- [Serve entry](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/apps/serve/main.cpp): `patches/changed-files/apps/serve/main.cpp`
- [Generation service](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/serve/generation_service.cpp): `src-tree/fusion-engine-src/src/serve/generation_service.cpp`

### Prompt and Generation

- [Engine API](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/include/ninfer/engine.h): `src-tree/fusion-engine-src/include/ninfer/engine.h`
- [Engine runtime](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/runtime/engine/engine.cpp): `patches/changed-files/src/runtime/engine/engine.cpp`
- [Prompt frontend](https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/frontend): `src-tree/fusion-engine-src/src/models/qwen3_5/frontend`
- [Qwen program](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/program.h): `patches/changed-files/src/models/qwen3_5/program/program.h`
- [Prompt prefill](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/prefill.cpp): `patches/changed-files/src/models/qwen3_5/program/prefill.cpp`

### Context Management

- [Decoder state](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/state/decoder_state.cpp): `patches/changed-files/src/models/qwen3_5/state/decoder_state.cpp`
- [Paged KV cache](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/core/paged_kv_cache.cpp): `patches/changed-files/src/core/paged_kv_cache.cpp`
- [Context storage](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/storage/context.cpp): `patches/changed-files/src/models/qwen3_5/program/storage/context.cpp`
- [Checkpoint capture](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/transactions/capture.cpp): `patches/changed-files/src/models/qwen3_5/program/transactions/capture.cpp`
- [Checkpoint materialization](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/transactions/materialization.cpp): `patches/changed-files/src/models/qwen3_5/program/transactions/materialization.cpp`
- [Pressure planning](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/patches/changed-files/src/models/qwen3_5/program/planning/pressure.cpp): `patches/changed-files/src/models/qwen3_5/program/planning/pressure.cpp`
- [KVMem retrieval](https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/ops): `src-tree/fusion-engine-src/src/ops`

### Model Execution

- [Qwen execution](https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/execution): `src-tree/fusion-engine-src/src/models/qwen3_5/execution`
- [Speculative decoding](https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/models/qwen3_5/program/speculative): `src-tree/fusion-engine-src/src/models/qwen3_5/program/speculative`
- [Vision and media](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/models/qwen3_5/execution/vision.cpp): `src-tree/fusion-engine-src/src/models/qwen3_5/execution/vision.cpp`

### Runtime Platform

- [CUDA device runtime](https://github.com/1314521gjy/ninfer-fusion-kvmem/blob/main/src-tree/fusion-engine-src/src/core/device.h): `src-tree/fusion-engine-src/src/core/device.h`
- [Device profiles](https://github.com/1314521gjy/ninfer-fusion-kvmem/tree/main/src-tree/fusion-engine-src/src/runtime/engine): `src-tree/fusion-engine-src/src/runtime/engine`

### Other components

- **Prompt user**

## Connections

- Prompt user → Serve entry: submits prompt (evidence: `README.md`)
- Serve entry → Generation service: starts service (evidence: `patches/changed-files/apps/serve/main.cpp`)
- Generation service → Engine API: serves generation
- Engine API → Engine runtime: delegates execution (evidence: `patches/changed-files/src/runtime/engine/engine.cpp`)
- Engine runtime → Prompt frontend: prepares prompt
- Engine runtime → Qwen program: runs generation
- Qwen program → Prompt prefill: prefills prompt (evidence: `README.md`)
- Qwen program → Decoder state: uses state
- Qwen program → Qwen execution: executes model
- Qwen execution → CUDA device runtime: uses CUDA
- Qwen program → Speculative decoding: supports speculation (evidence: `README.md`)
- Qwen program → Vision and media: supports vision (evidence: `README.md`)
- Context storage → Paged KV cache: manages KV (evidence: `README.md`)
- Checkpoint capture → Context storage: captures checkpoints (evidence: `README.md`)
- Pressure planning → Checkpoint materialization: plans materialization (evidence: `README.md`)
- Checkpoint materialization → Context storage: materializes context (evidence: `README.md`)
- Prompt prefill → KVMem retrieval: retrieves relevant pages (evidence: `README.md`)
- Engine runtime → Device profiles: selects device profile (evidence: `README.md`)
