How tiiny-ai/powerinfer works
PowerInfer is a local large-language-model inference engine designed to combine CPU and consumer-GPU resources using activation locality and sparse computation. A typical workflow prepares model weights in PowerInfer GGUF, loads them into an inference context, and generates tokens through a CLI or server; the server can stream chat responses to its web UI, which stores conversations locally. The graph includes documented conversion and inference capabilities alongside sampled runtime components. Several implementation links are omitted or dashed because the excerpts do not show their dispatch paths.
Main entry points: llama-context.cpp Inference context · ggml.cpp Tensor runtime · app.context.tsx Chat state · server.cpp HTTP server
Components 17
Model preparation
- HF converter
smallthinker/convert_hf_to_gguf.py - GGUF tooling
smallthinker/gguf-py/gguf/gguf_writer.py - Model weights
Inference runtime
- Inference context
smallthinker/src/llama-context.cpp - Llama API
smallthinker/include/llama.h - Tensor runtime
smallthinker/ggml/src/ggml.cpp - CPU kernels
smallthinker/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp - GPU backends
smallthinker/ggml/src/ggml-cuda
Sparse execution
- MoE pipeline
smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp - Expert cache
smallthinker/powerinfer/moe_sparse_pipeline/moe_sparse_pipeline
Chat serving
- Interactive CLI
smallthinker/tools/main/main.cpp - HTTP server
smallthinker/tools/server/server.cpp
Web client
- Chat interface
smallthinker/tools/server/webui/src/App.tsx - Chat state
smallthinker/tools/server/webui/src/utils/app.context.tsx - Conversation store
smallthinker/tools/server/webui/src/utils/storage.ts
Other components
- User
- Browser user
Connections 17
- User → HF converter: prepares weights
- HF converter → GGUF tooling: uses GGUF
smallthinker/convert_hf_to_gguf.py - GGUF tooling → Model weights: writes format
- User → Interactive CLI: runs inference
smallthinker/tools/main/main.cpp - Interactive CLI → Inference context: initializes model
smallthinker/tools/main/main.cpp - User → HTTP server: submits chat
- HTTP server → Inference context: loads model
smallthinker/tools/server/server.cpp - Llama API → Inference context: exposes inference
smallthinker/include/llama.h - Inference context → Tensor runtime: computes graphs
smallthinker/src/llama-context.cpp - Tensor runtime → CPU kernels: dispatches kernels
- Tensor runtime → GPU backends: dispatches kernels
- Inference context → MoE pipeline: invokes sparse path
- MoE pipeline → Expert cache: fetches experts
smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp - Browser user → Chat interface: uses chat
smallthinker/tools/server/webui/src/App.tsx - Chat interface → Chat state: renders state
smallthinker/tools/server/webui/src/App.tsx - Chat state → HTTP server: streams requests
smallthinker/tools/server/webui/src/utils/app.context.tsx - Chat state → Conversation store: loads conversations
smallthinker/tools/server/webui/src/utils/app.context.tsx