# liujun-7788/ninfer-v3-v100 architecture

> Architecture diagram of the GitHub repository liujun-7788/ninfer-v3-v100, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liujun-7788/ninfer-v3-v100
- Repository: https://github.com/liujun-7788/ninfer-v3-v100

## Overview

NInfer is a CUDA inference engine for selected Qwen models, exposed through a local CLI and OpenAI-/Anthropic-compatible HTTP APIs. The main workflow accepts text or media, prepares a model-specific prompt, schedules prefill and autoregressive/speculative decoding, and returns generated output. Artifacts supply weights and frontend resources; the runtime maintains KV and other request state while GPU operators execute model work. The README also documents perplexity scoring and MTP/DFlash speculative decoding. Some internal wiring is inferred only where the sampled source or README supports it; unsampled subsystem links are omitted.

Main entry points: [`src/targets/qwen3_6/impl/runtime/program.h`](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/runtime/program.h) (Model target runtime), [`src/runtime/engine/engine.cpp`](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine.cpp) (Engine API), [`src/runtime/engine/engine_core.h`](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine_core.h) (Request scheduler), [`src/serve/generation_service.cpp`](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/generation_service.cpp) (Generation service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_cli["Local CLI<br/>[main.cpp]"]
  node_server["HTTP server<br/>[http_server.cpp]"]
  node_api["API protocols"]
  node_anthropic["Anthropic protocol"]
  node_service["Generation service"]
end

subgraph group_request["Request preparation"]
  node_prompt["Prompt input<br/>[prompt_input.cpp]"]
  node_media["Media acquisition<br/>[acquire.cpp]"]
  node_frontend["Qwen frontend<br/>[frontend.cpp]"]
end

subgraph group_engine["Inference engine"]
  node_engine_api["Engine API<br/>[engine.cpp]"]
  node_scheduler["Request scheduler<br/>[engine_core.h]"]
  node_resources["Resource planning<br/>[resource_manager.h]"]
  node_kv[("KV cache state<br/>[paged_kv_cache.cpp]")]
end

subgraph group_model["Model execution"]
  node_target["Model target runtime<br/>[program.h]"]
  node_ops["CUDA operators"]
  node_speculative["Speculative decoding"]
  node_sampling["Sampling and output<br/>[sampling.cpp]"]
end

subgraph group_artifacts["Artifacts and scoring"]
  node_artifact["Artifact loading<br/>[reader.cpp]"]
  node_targets["Model variants<br/>[registry.cpp]"]
  node_perplexity["Perplexity scoring<br/>[main.cpp]"]
end

node_user(("User or client"))

node_user -->|"submits prompt"| node_cli
node_user -->|"sends HTTP request"| node_server
node_cli -->|"prepares input"| node_prompt
node_cli -->|"requests generation"| node_engine_api
node_server -->|"dispatches protocol"| node_api
node_server -->|"dispatches protocol"| node_anthropic
node_api -->|"submits request"| node_service
node_anthropic -->|"submits request"| node_service
node_service -->|"acquires media"| node_media
node_service -->|"prepares and generates"| node_engine_api
node_prompt -->|"provides prompt"| node_frontend
node_engine_api -->|"loads artifact"| node_artifact
node_engine_api -->|"constructs target"| node_targets
node_engine_api -->|"submits work"| node_scheduler
node_scheduler -->|"plans capacity"| node_resources
node_scheduler -->|"advances prefill"| node_target
node_scheduler -->|"advances decode"| node_target
node_target -->|"uses tokenizer and processor"| node_frontend
node_target -->|"launches operators"| node_ops
node_target -->|"reads and writes cache"| node_kv
node_target -.->|"uses draft decoding"| node_speculative
node_target -->|"samples output"| node_sampling
node_perplexity -->|"scores token logprobs"| node_engine_api

click node_cli "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/apps/cli/main.cpp"
click node_server "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/http_server.cpp"
click node_api "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/openai_chat_http.cpp"
click node_anthropic "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/anthropic_messages_http.cpp"
click node_service "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/generation_service.cpp"
click node_prompt "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/product/prompt_input/prompt_input.cpp"
click node_media "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/product/media_acquire/acquire.cpp"
click node_frontend "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/frontend/frontend.cpp"
click node_engine_api "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine.cpp"
click node_scheduler "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine_core.h"
click node_resources "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/resource_manager.h"
click node_kv "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/core/paged_kv_cache.cpp"
click node_target "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/runtime/program.h"
click node_speculative "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/runtime/speculative_target_impl.h"
click node_sampling "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/contract/sampling.cpp"
click node_artifact "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/artifact/reader.cpp"
click node_targets "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/registry.cpp"
click node_perplexity "https://github.com/liujun-7788/ninfer-v3-v100/blob/master/apps/perplexity/main.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_server,node_api,node_anthropic,node_service,node_user toneBlue
class node_prompt,node_media,node_frontend toneAmber
class node_engine_api,node_scheduler,node_resources,node_kv toneMint
class node_target,node_ops,node_speculative,node_sampling toneRose
class node_artifact,node_targets,node_perplexity toneIndigo
```

## Components

### User interfaces

- [Local CLI](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/apps/cli/main.cpp): `apps/cli/main.cpp`
- [HTTP server](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/http_server.cpp): `src/serve/http_server.cpp`
- [API protocols](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/openai_chat_http.cpp): `src/serve/openai_chat_http.cpp`
- [Anthropic protocol](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/anthropic_messages_http.cpp): `src/serve/anthropic_messages_http.cpp`
- [Generation service](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/serve/generation_service.cpp): `src/serve/generation_service.cpp`

### Request preparation

- [Prompt input](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/product/prompt_input/prompt_input.cpp): `src/product/prompt_input/prompt_input.cpp`
- [Media acquisition](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/product/media_acquire/acquire.cpp): `src/product/media_acquire/acquire.cpp`
- [Qwen frontend](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/frontend/frontend.cpp): `src/targets/qwen3_6/impl/frontend/frontend.cpp`

### Inference engine

- [Engine API](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine.cpp): `src/runtime/engine/engine.cpp`
- [Request scheduler](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/engine_core.h): `src/runtime/engine/engine_core.h`
- [Resource planning](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/engine/resource_manager.h): `src/runtime/engine/resource_manager.h`
- [KV cache state](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/core/paged_kv_cache.cpp): `src/core/paged_kv_cache.cpp`

### Model execution

- [Model target runtime](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/runtime/program.h): `src/targets/qwen3_6/impl/runtime/program.h`
- **CUDA operators**
- [Speculative decoding](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/qwen3_6/impl/runtime/speculative_target_impl.h): `src/targets/qwen3_6/impl/runtime/speculative_target_impl.h`
- [Sampling and output](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/runtime/contract/sampling.cpp): `src/runtime/contract/sampling.cpp`

### Artifacts and scoring

- [Artifact loading](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/artifact/reader.cpp): `src/artifact/reader.cpp`
- [Model variants](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/src/targets/registry.cpp): `src/targets/registry.cpp`
- [Perplexity scoring](https://github.com/liujun-7788/ninfer-v3-v100/blob/master/apps/perplexity/main.cpp): `apps/perplexity/main.cpp`

### Other components

- **User or client**

## Connections

- User or client → Local CLI: submits prompt
- User or client → HTTP server: sends HTTP request
- Local CLI → Prompt input: prepares input
- Local CLI → Engine API: requests generation
- HTTP server → API protocols: dispatches protocol
- HTTP server → Anthropic protocol: dispatches protocol
- API protocols → Generation service: submits request
- Anthropic protocol → Generation service: submits request
- Generation service → Media acquisition: acquires media
- Generation service → Engine API: prepares and generates
- Prompt input → Qwen frontend: provides prompt
- Engine API → Artifact loading: loads artifact
- Engine API → Model variants: constructs target
- Engine API → Request scheduler: submits work
- Request scheduler → Resource planning: plans capacity
- Request scheduler → Model target runtime: advances prefill
- Request scheduler → Model target runtime: advances decode
- Model target runtime → Qwen frontend: uses tokenizer and processor
- Model target runtime → CUDA operators: launches operators
- Model target runtime → KV cache state: reads and writes cache
- Model target runtime → Speculative decoding: uses draft decoding
- Model target runtime → Sampling and output: samples output
- Perplexity scoring → Engine API: scores token logprobs