How 5258mf/ninfer-rtx3060-27b works
NInfer is a Windows-focused local inference engine for Qwen models, with Bonsai2 and Swift engine variants. The main workflow is a client request entering an HTTP-compatible API, being translated and enriched with optional media, then queued and executed by the engine to produce a response. KVMem supports long-context retrieval; model artifacts, GPU operations, logging, and perplexity evaluation are supporting capabilities. Some subsystem links below are documented rather than verified from sampled implementation, which is marked dashed or omitted.
Main entry points: generation_service.cpp Generation service · engine.cpp Engine API · program.h Qwen runtime · engine_core.h Request execution
Components 19
Serving
- Serve application
bonsai2/engine/apps/serve/main.cpp - HTTP server
bonsai2/engine/src/serve/http_server.cpp - API adapters
bonsai2/engine/src/serve/openai_chat_http.cpp - Generation service
bonsai2/engine/src/serve/generation_service.cpp - Request translation
bonsai2/engine/src/serve/translate.cpp - Media acquisition
bonsai2/engine/src/product/media_acquire/acquire.cpp
Inference runtime
- Engine API
bonsai2/engine/src/runtime/engine/engine.cpp - Request execution
bonsai2/engine/src/runtime/engine/engine_core.h - Resource planning
bonsai2/engine/src/runtime/engine/resource_manager.h - CUDA device
bonsai2/engine/src/core/device.h - GPU operations
bonsai2/engine/src/ops/wrapper/linear_swiglu.cpp
Model and context
- Qwen runtime
bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h - KVMem retrieval
bonsai2/engine/src/ops/kvmem/kvmem_retrieve.cpp - Paged KV storage
bonsai2/engine/src/core/paged_kv_storage.h - Artifact loading
bonsai2/engine/src/artifact/reader.cpp
Supporting capabilities
- Interactive CLI
bonsai2/engine/apps/cli/main.cpp - Perplexity evaluation
bonsai2/engine/apps/perplexity/evaluation.cpp - Runtime logging
bonsai2/engine/src/product/logging/logging.cpp
Other components
- API client
Connections 17
- API client → HTTP server: sends requests
README.md - Serve application → HTTP server: starts server
bonsai2/engine/apps/serve/main.cpp - Serve application → Generation service: starts service
bonsai2/engine/apps/serve/main.cpp - HTTP server → API adapters: routes requests
- API adapters → Generation service: submits requests
- Generation service → Request translation: uses translation
bonsai2/engine/src/serve/generation_service.cpp - Generation service → Media acquisition: acquires media
bonsai2/engine/src/serve/generation_service.cpp - Generation service → Engine API: requests generation
- Engine API → Request execution: delegates execution
bonsai2/engine/src/runtime/engine/engine.cpp - Request execution → Resource planning: plans resources
bonsai2/engine/src/runtime/engine/engine_core.h - Request execution → Qwen runtime: executes sequences
- Qwen runtime → KVMem retrieval: uses retrieval
bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h - Qwen runtime → Paged KV storage: uses KV storage
- Qwen runtime → GPU operations: runs operations
- Engine API → Artifact loading: loads model
- Interactive CLI → Engine API: runs inference
bonsai2/engine/apps/cli/main.cpp - Perplexity evaluation → Engine API: scores tokens