How 1314521gjy/ninfer-fusion-kvmem works
NInfer is a CUDA inference engine and serving application, with downstream changes aimed at long-context generation on a smaller device KV pool. The principal workflow accepts a prompt, prepares it for generation, plans and runs model work, and reuses or retrieves context state as generation proceeds. The README documents content-ranked KVMem retrieval, host-backed checkpoint reuse, adaptive device profiles, speculative decoding, and multimodal support. Several subsystem links are documented at a high level; unsampled implementation details are not treated as verified call paths.
Main entry points: program.h Qwen program · engine.cpp Engine runtime · context.cpp Context storage · main.cpp Serve entry
Components 20
Serving and API
- Serve entry
patches/changed-files/apps/serve/main.cpp - Generation service
src-tree/fusion-engine-src/src/serve/generation_service.cpp
Prompt and Generation
- Engine API
src-tree/fusion-engine-src/include/ninfer/engine.h - Engine runtime
patches/changed-files/src/runtime/engine/engine.cpp - Prompt frontend
src-tree/fusion-engine-src/src/models/qwen3_5/frontend - Qwen program
patches/changed-files/src/models/qwen3_5/program/program.h - Prompt prefill
patches/changed-files/src/models/qwen3_5/program/prefill.cpp
Context Management
- Decoder state
patches/changed-files/src/models/qwen3_5/state/decoder_state.cpp - Paged KV cache
patches/changed-files/src/core/paged_kv_cache.cpp - Context storage
patches/changed-files/src/models/qwen3_5/program/storage/context.cpp - Checkpoint capture
patches/changed-files/src/models/qwen3_5/program/transactions/capture.cpp - Checkpoint materialization
patches/changed-files/src/models/qwen3_5/program/transactions/materialization.cpp - Pressure planning
patches/changed-files/src/models/qwen3_5/program/planning/pressure.cpp - KVMem retrieval
src-tree/fusion-engine-src/src/ops
Model Execution
- Qwen execution
src-tree/fusion-engine-src/src/models/qwen3_5/execution - Speculative decoding
src-tree/fusion-engine-src/src/models/qwen3_5/program/speculative - Vision and media
src-tree/fusion-engine-src/src/models/qwen3_5/execution/vision.cpp
Runtime Platform
- CUDA device runtime
src-tree/fusion-engine-src/src/core/device.h - Device profiles
src-tree/fusion-engine-src/src/runtime/engine
Other components
- Prompt user
Connections 18
- Prompt user → Serve entry: submits prompt
README.md - Serve entry → Generation service: starts service
patches/changed-files/apps/serve/main.cpp - Generation service → Engine API: serves generation
- Engine API → Engine runtime: delegates execution
patches/changed-files/src/runtime/engine/engine.cpp - Engine runtime → Prompt frontend: prepares prompt
- Engine runtime → Qwen program: runs generation
- Qwen program → Prompt prefill: prefills prompt
README.md - Qwen program → Decoder state: uses state
- Qwen program → Qwen execution: executes model
- Qwen execution → CUDA device runtime: uses CUDA
- Qwen program → Speculative decoding: supports speculation
README.md - Qwen program → Vision and media: supports vision
README.md - Context storage → Paged KV cache: manages KV
README.md - Checkpoint capture → Context storage: captures checkpoints
README.md - Pressure planning → Checkpoint materialization: plans materialization
README.md - Checkpoint materialization → Context storage: materializes context
README.md - Prompt prefill → KVMem retrieval: retrieves relevant pages
README.md - Engine runtime → Device profiles: selects device profile
README.md