# 5258mf/ninfer-rtx3060-27b architecture

> Architecture diagram of the GitHub repository 5258mf/ninfer-rtx3060-27b, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/5258mf/ninfer-rtx3060-27b
- Repository: https://github.com/5258mf/ninfer-rtx3060-27b

## Overview

NInfer is a Windows-focused local inference engine for Qwen models, with Bonsai2 and Swift engine variants. The main workflow is a client request entering an HTTP-compatible API, being translated and enriched with optional media, then queued and executed by the engine to produce a response. KVMem supports long-context retrieval; model artifacts, GPU operations, logging, and perplexity evaluation are supporting capabilities. Some subsystem links below are documented rather than verified from sampled implementation, which is marked dashed or omitted.

Main entry points: [`bonsai2/engine/src/serve/generation_service.cpp`](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/generation_service.cpp) (Generation service), [`bonsai2/engine/src/runtime/engine/engine.cpp`](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine.cpp) (Engine API), [`bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h`](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h) (Qwen runtime), [`bonsai2/engine/src/runtime/engine/engine_core.h`](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine_core.h) (Request execution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving"]
  node_serve_app["Serve application<br/>[main.cpp]"]
  node_http["HTTP server<br/>[http_server.cpp]"]
  node_protocols["API adapters"]
  node_generation["Generation service"]
  node_translation["Request translation<br/>[translate.cpp]"]
  node_media["Media acquisition<br/>[acquire.cpp]"]
end

subgraph group_runtime["Inference runtime"]
  node_engine_api["Engine API<br/>[engine.cpp]"]
  node_engine_core["Request execution<br/>[engine_core.h]"]
  node_resources["Resource planning<br/>[resource_manager.h]"]
  node_device["CUDA device<br/>[device.h]"]
  node_ops["GPU operations<br/>[linear_swiglu.cpp]"]
end

subgraph group_model["Model and context"]
  node_model_runtime["Qwen runtime<br/>[program.h]"]
  node_kvmem["KVMem retrieval<br/>[kvmem_retrieve.cpp]"]
  node_kv_cache[("Paged KV storage<br/>[paged_kv_storage.h]")]
  node_artifacts["Artifact loading<br/>[reader.cpp]"]
end

subgraph group_support["Supporting capabilities"]
  node_cli["Interactive CLI<br/>[main.cpp]"]
  node_perplexity["Perplexity evaluation<br/>[evaluation.cpp]"]
  node_logging["Runtime logging<br/>[logging.cpp]"]
end

node_client(("API client"))

node_client -->|"sends requests"| node_http
node_serve_app -->|"starts server"| node_http
node_serve_app -->|"starts service"| node_generation
node_http -.->|"routes requests"| node_protocols
node_protocols -.->|"submits requests"| node_generation
node_generation -->|"uses translation"| node_translation
node_generation -->|"acquires media"| node_media
node_generation -.->|"requests generation"| node_engine_api
node_engine_api -->|"delegates execution"| node_engine_core
node_engine_core -->|"plans resources"| node_resources
node_engine_core -.->|"executes sequences"| node_model_runtime
node_model_runtime -->|"uses retrieval"| node_kvmem
node_model_runtime -.->|"uses KV storage"| node_kv_cache
node_model_runtime -.->|"runs operations"| node_ops
node_engine_api -.->|"loads model"| node_artifacts
node_cli -->|"runs inference"| node_engine_api
node_perplexity -->|"scores tokens"| node_engine_api

click node_serve_app "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/serve/main.cpp"
click node_http "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/http_server.cpp"
click node_protocols "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/openai_chat_http.cpp"
click node_generation "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/generation_service.cpp"
click node_translation "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/translate.cpp"
click node_media "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/product/media_acquire/acquire.cpp"
click node_engine_api "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine.cpp"
click node_engine_core "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine_core.h"
click node_resources "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/resource_manager.h"
click node_device "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/core/device.h"
click node_model_runtime "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h"
click node_kvmem "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/ops/kvmem/kvmem_retrieve.cpp"
click node_kv_cache "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/core/paged_kv_storage.h"
click node_artifacts "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/artifact/reader.cpp"
click node_ops "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/ops/wrapper/linear_swiglu.cpp"
click node_cli "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/cli/main.cpp"
click node_perplexity "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/perplexity/evaluation.cpp"
click node_logging "https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/product/logging/logging.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_serve_app,node_http,node_protocols,node_generation,node_translation,node_media,node_client toneBlue
class node_engine_api,node_engine_core,node_resources,node_device,node_ops toneAmber
class node_model_runtime,node_kvmem,node_kv_cache,node_artifacts toneMint
class node_cli,node_perplexity,node_logging toneRose
```

## Components

### Serving

- [Serve application](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/serve/main.cpp): `bonsai2/engine/apps/serve/main.cpp`
- [HTTP server](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/http_server.cpp): `bonsai2/engine/src/serve/http_server.cpp`
- [API adapters](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/openai_chat_http.cpp): `bonsai2/engine/src/serve/openai_chat_http.cpp`
- [Generation service](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/generation_service.cpp): `bonsai2/engine/src/serve/generation_service.cpp`
- [Request translation](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/serve/translate.cpp): `bonsai2/engine/src/serve/translate.cpp`
- [Media acquisition](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/product/media_acquire/acquire.cpp): `bonsai2/engine/src/product/media_acquire/acquire.cpp`

### Inference runtime

- [Engine API](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine.cpp): `bonsai2/engine/src/runtime/engine/engine.cpp`
- [Request execution](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/engine_core.h): `bonsai2/engine/src/runtime/engine/engine_core.h`
- [Resource planning](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/runtime/engine/resource_manager.h): `bonsai2/engine/src/runtime/engine/resource_manager.h`
- [CUDA device](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/core/device.h): `bonsai2/engine/src/core/device.h`
- [GPU operations](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/ops/wrapper/linear_swiglu.cpp): `bonsai2/engine/src/ops/wrapper/linear_swiglu.cpp`

### Model and context

- [Qwen runtime](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h): `bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h`
- [KVMem retrieval](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/ops/kvmem/kvmem_retrieve.cpp): `bonsai2/engine/src/ops/kvmem/kvmem_retrieve.cpp`
- [Paged KV storage](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/core/paged_kv_storage.h): `bonsai2/engine/src/core/paged_kv_storage.h`
- [Artifact loading](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/artifact/reader.cpp): `bonsai2/engine/src/artifact/reader.cpp`

### Supporting capabilities

- [Interactive CLI](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/cli/main.cpp): `bonsai2/engine/apps/cli/main.cpp`
- [Perplexity evaluation](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/apps/perplexity/evaluation.cpp): `bonsai2/engine/apps/perplexity/evaluation.cpp`
- [Runtime logging](https://github.com/5258mf/ninfer-rtx3060-27b/blob/main/bonsai2/engine/src/product/logging/logging.cpp): `bonsai2/engine/src/product/logging/logging.cpp`

### Other components

- **API client**

## Connections

- API client → HTTP server: sends requests (evidence: `README.md`)
- Serve application → HTTP server: starts server (evidence: `bonsai2/engine/apps/serve/main.cpp`)
- Serve application → Generation service: starts service (evidence: `bonsai2/engine/apps/serve/main.cpp`)
- HTTP server → API adapters: routes requests
- API adapters → Generation service: submits requests
- Generation service → Request translation: uses translation (evidence: `bonsai2/engine/src/serve/generation_service.cpp`)
- Generation service → Media acquisition: acquires media (evidence: `bonsai2/engine/src/serve/generation_service.cpp`)
- Generation service → Engine API: requests generation
- Engine API → Request execution: delegates execution (evidence: `bonsai2/engine/src/runtime/engine/engine.cpp`)
- Request execution → Resource planning: plans resources (evidence: `bonsai2/engine/src/runtime/engine/engine_core.h`)
- Request execution → Qwen runtime: executes sequences
- Qwen runtime → KVMem retrieval: uses retrieval (evidence: `bonsai2/engine/src/targets/qwen3_6/impl/runtime/program.h`)
- Qwen runtime → Paged KV storage: uses KV storage
- Qwen runtime → GPU operations: runs operations
- Engine API → Artifact loading: loads model
- Interactive CLI → Engine API: runs inference (evidence: `bonsai2/engine/apps/cli/main.cpp`)
- Perplexity evaluation → Engine API: scores tokens
