How liujun-7788/ninfer-v3-v100 works
NInfer is a CUDA inference engine for selected Qwen models, exposed through a local CLI and OpenAI-/Anthropic-compatible HTTP APIs. The main workflow accepts text or media, prepares a model-specific prompt, schedules prefill and autoregressive/speculative decoding, and returns generated output. Artifacts supply weights and frontend resources; the runtime maintains KV and other request state while GPU operators execute model work. The README also documents perplexity scoring and MTP/DFlash speculative decoding. Some internal wiring is inferred only where the sampled source or README supports it; unsampled subsystem links are omitted.
Main entry points: program.h Model target runtime · engine.cpp Engine API · engine_core.h Request scheduler · generation_service.cpp Generation service
Components 20
User interfaces
- Local CLI
apps/cli/main.cpp - HTTP server
src/serve/http_server.cpp - API protocols
src/serve/openai_chat_http.cpp - Anthropic protocol
src/serve/anthropic_messages_http.cpp - Generation service
src/serve/generation_service.cpp
Request preparation
- Prompt input
src/product/prompt_input/prompt_input.cpp - Media acquisition
src/product/media_acquire/acquire.cpp - Qwen frontend
src/targets/qwen3_6/impl/frontend/frontend.cpp
Inference engine
- Engine API
src/runtime/engine/engine.cpp - Request scheduler
src/runtime/engine/engine_core.h - Resource planning
src/runtime/engine/resource_manager.h - KV cache state
src/core/paged_kv_cache.cpp
Model execution
- Model target runtime
src/targets/qwen3_6/impl/runtime/program.h - CUDA operators
- Speculative decoding
src/targets/qwen3_6/impl/runtime/speculative_target_impl.h - Sampling and output
src/runtime/contract/sampling.cpp
Artifacts and scoring
- Artifact loading
src/artifact/reader.cpp - Model variants
src/targets/registry.cpp - Perplexity scoring
apps/perplexity/main.cpp
Other components
- User or client
Connections 23
- User or client → Local CLI: submits prompt
- User or client → HTTP server: sends HTTP request
- Local CLI → Prompt input: prepares input
- Local CLI → Engine API: requests generation
- HTTP server → API protocols: dispatches protocol
- HTTP server → Anthropic protocol: dispatches protocol
- API protocols → Generation service: submits request
- Anthropic protocol → Generation service: submits request
- Generation service → Media acquisition: acquires media
- Generation service → Engine API: prepares and generates
- Prompt input → Qwen frontend: provides prompt
- Engine API → Artifact loading: loads artifact
- Engine API → Model variants: constructs target
- Engine API → Request scheduler: submits work
- Request scheduler → Resource planning: plans capacity
- Request scheduler → Model target runtime: advances prefill
- Request scheduler → Model target runtime: advances decode
- Model target runtime → Qwen frontend: uses tokenizer and processor
- Model target runtime → CUDA operators: launches operators
- Model target runtime → KV cache state: reads and writes cache
- Model target runtime → Speculative decoding: uses draft decoding
- Model target runtime → Sampling and output: samples output
- Perplexity scoring → Engine API: scores token logprobs