# tiiny-ai/powerinfer architecture

> Architecture diagram of the GitHub repository tiiny-ai/powerinfer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tiiny-ai/powerinfer
- Repository: https://github.com/tiiny-ai/powerinfer

## Overview

PowerInfer is a local large-language-model inference engine designed to combine CPU and consumer-GPU resources using activation locality and sparse computation. A typical workflow prepares model weights in PowerInfer GGUF, loads them into an inference context, and generates tokens through a CLI or server; the server can stream chat responses to its web UI, which stores conversations locally. The graph includes documented conversion and inference capabilities alongside sampled runtime components. Several implementation links are omitted or dashed because the excerpts do not show their dispatch paths.

Main entry points: [`smallthinker/src/llama-context.cpp`](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/src/llama-context.cpp) (Inference context), [`smallthinker/ggml/src/ggml.cpp`](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml.cpp) (Tensor runtime), [`smallthinker/tools/server/webui/src/utils/app.context.tsx`](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/utils/app.context.tsx) (Chat state), [`smallthinker/tools/server/server.cpp`](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/server.cpp) (HTTP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_model_prep["Model preparation"]
  node_converter["HF converter"]
  node_gguf["GGUF tooling<br/>[gguf_writer.py]"]
  node_weights["Model weights"]
end

subgraph group_inference["Inference runtime"]
  node_context["Inference context<br/>[llama-context.cpp]"]
  node_api["Llama API<br/>[llama.h]"]
  node_ggml["Tensor runtime<br/>[ggml.cpp]"]
  node_cpu["CPU kernels"]
  node_gpu["GPU backends"]
end

subgraph group_sparse["Sparse execution"]
  node_moe["MoE pipeline<br/>[pipeline.cpp]"]
  node_expert_cache[("Expert cache")]
end

subgraph group_serving["Chat serving"]
  node_cli["Interactive CLI<br/>[main.cpp]"]
  node_server["HTTP server<br/>[server.cpp]"]
end

subgraph group_web["Web client"]
  node_webui["Chat interface<br/>[App.tsx]"]
  node_chat_state["Chat state<br/>[app.context.tsx]"]
  node_storage[("Conversation store<br/>[storage.ts]")]
end

node_user(("User"))
node_browser(("Browser user"))

node_user -.->|"prepares weights"| node_converter
node_converter -->|"uses GGUF"| node_gguf
node_gguf -.->|"writes format"| node_weights
node_user -->|"runs inference"| node_cli
node_cli -->|"initializes model"| node_context
node_user -.->|"submits chat"| node_server
node_server -->|"loads model"| node_context
node_api -.->|"exposes inference"| node_context
node_context -->|"computes graphs"| node_ggml
node_ggml -.->|"dispatches kernels"| node_cpu
node_ggml -.->|"dispatches kernels"| node_gpu
node_context -.->|"invokes sparse path"| node_moe
node_moe -->|"fetches experts"| node_expert_cache
node_browser -->|"uses chat"| node_webui
node_webui -->|"renders state"| node_chat_state
node_chat_state -->|"streams requests"| node_server
node_chat_state -->|"loads conversations"| node_storage

click node_converter "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/convert_hf_to_gguf.py"
click node_gguf "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/gguf-py/gguf/gguf_writer.py"
click node_cli "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/main/main.cpp"
click node_server "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/server.cpp"
click node_context "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/src/llama-context.cpp"
click node_api "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/include/llama.h"
click node_ggml "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml.cpp"
click node_cpu "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp"
click node_gpu "https://github.com/tiiny-ai/powerinfer/tree/main/smallthinker/ggml/src/ggml-cuda"
click node_moe "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp"
click node_expert_cache "https://github.com/tiiny-ai/powerinfer/tree/main/smallthinker/powerinfer/moe_sparse_pipeline/moe_sparse_pipeline"
click node_webui "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/App.tsx"
click node_chat_state "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/utils/app.context.tsx"
click node_storage "https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/utils/storage.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_converter,node_gguf,node_weights,node_user,node_browser toneBlue
class node_context,node_api,node_ggml,node_cpu,node_gpu toneAmber
class node_moe,node_expert_cache toneMint
class node_cli,node_server toneRose
class node_webui,node_chat_state,node_storage toneIndigo
```

## Components

### Model preparation

- [HF converter](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/convert_hf_to_gguf.py): `smallthinker/convert_hf_to_gguf.py`
- [GGUF tooling](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/gguf-py/gguf/gguf_writer.py): `smallthinker/gguf-py/gguf/gguf_writer.py`
- **Model weights**

### Inference runtime

- [Inference context](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/src/llama-context.cpp): `smallthinker/src/llama-context.cpp`
- [Llama API](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/include/llama.h): `smallthinker/include/llama.h`
- [Tensor runtime](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml.cpp): `smallthinker/ggml/src/ggml.cpp`
- [CPU kernels](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp): `smallthinker/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp`
- [GPU backends](https://github.com/tiiny-ai/powerinfer/tree/main/smallthinker/ggml/src/ggml-cuda): `smallthinker/ggml/src/ggml-cuda`

### Sparse execution

- [MoE pipeline](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp): `smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp`
- [Expert cache](https://github.com/tiiny-ai/powerinfer/tree/main/smallthinker/powerinfer/moe_sparse_pipeline/moe_sparse_pipeline): `smallthinker/powerinfer/moe_sparse_pipeline/moe_sparse_pipeline`

### Chat serving

- [Interactive CLI](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/main/main.cpp): `smallthinker/tools/main/main.cpp`
- [HTTP server](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/server.cpp): `smallthinker/tools/server/server.cpp`

### Web client

- [Chat interface](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/App.tsx): `smallthinker/tools/server/webui/src/App.tsx`
- [Chat state](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/utils/app.context.tsx): `smallthinker/tools/server/webui/src/utils/app.context.tsx`
- [Conversation store](https://github.com/tiiny-ai/powerinfer/blob/main/smallthinker/tools/server/webui/src/utils/storage.ts): `smallthinker/tools/server/webui/src/utils/storage.ts`

### Other components

- **User**
- **Browser user**

## Connections

- User → HF converter: prepares weights
- HF converter → GGUF tooling: uses GGUF (evidence: `smallthinker/convert_hf_to_gguf.py`)
- GGUF tooling → Model weights: writes format
- User → Interactive CLI: runs inference (evidence: `smallthinker/tools/main/main.cpp`)
- Interactive CLI → Inference context: initializes model (evidence: `smallthinker/tools/main/main.cpp`)
- User → HTTP server: submits chat
- HTTP server → Inference context: loads model (evidence: `smallthinker/tools/server/server.cpp`)
- Llama API → Inference context: exposes inference (evidence: `smallthinker/include/llama.h`)
- Inference context → Tensor runtime: computes graphs (evidence: `smallthinker/src/llama-context.cpp`)
- Tensor runtime → CPU kernels: dispatches kernels
- Tensor runtime → GPU backends: dispatches kernels
- Inference context → MoE pipeline: invokes sparse path
- MoE pipeline → Expert cache: fetches experts (evidence: `smallthinker/powerinfer/moe_sparse_pipeline/pipeline.cpp`)
- Browser user → Chat interface: uses chat (evidence: `smallthinker/tools/server/webui/src/App.tsx`)
- Chat interface → Chat state: renders state (evidence: `smallthinker/tools/server/webui/src/App.tsx`)
- Chat state → HTTP server: streams requests (evidence: `smallthinker/tools/server/webui/src/utils/app.context.tsx`)
- Chat state → Conversation store: loads conversations (evidence: `smallthinker/tools/server/webui/src/utils/app.context.tsx`)
