# wici-ai/inferssd architecture

> Architecture diagram of the GitHub repository wici-ai/inferssd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wici-ai/inferssd
- Repository: https://github.com/wici-ai/inferssd

## Overview

InferSSD runs large GGUF mixture-of-experts models by placing expert weights on SSD while performing inference through its llama.cpp-derived engine. The user selects a model and starts CLI chat or the server; model acquisition and machine checks precede loading, then inference produces generated tokens for the chosen interface. The repository also includes a browser chat UI, OpenAI-compatible endpoints, and ordinary RAM inference for Gemma. Several launcher and storage internals are unsampled, so those relationships are shown only where documented or explicitly marked unverified.

Main entry points: [`framework/src/inferssd/app/command.py`](https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/app/command.py) (CLI launcher), [`engine/src/llama-context.cpp`](https://github.com/wici-ai/inferssd/blob/main/engine/src/llama-context.cpp) (Inference context), [`engine/tools/server/server.cpp`](https://github.com/wici-ai/inferssd/blob/main/engine/tools/server/server.cpp) (Inference server), [`engine/src/ssd/cpu-runtime.h`](https://github.com/wici-ai/inferssd/blob/main/engine/src/ssd/cpu-runtime.h) (SSD runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_launch["Launch and models"]
  node_launcher["CLI launcher<br/>[command.py]"]
  node_machine["Machine checks<br/>[identity.py]"]
  node_model_source["Model source<br/>[gguf_alias.py]"]
  node_model_store[("GGUF model store<br/>[index.cpp]")]
end

subgraph group_inference["Inference engine"]
  node_model_loader["Model loading<br/>[llama-context.cpp]"]
  node_model_graphs["Model architectures<br/>[models.h]"]
  node_llama_api["Llama API<br/>[llama.h]"]
  node_context["Inference context<br/>[llama-context.cpp]"]
  node_tensor_ops["Tensor operations<br/>[ggml.h]"]
  node_backend["Compute backends<br/>[ggml-backend.h]"]
  node_common["Shared inference support<br/>[common.h]"]
end

subgraph group_ssd["SSD execution"]
  node_ssd_runtime["SSD runtime<br/>[cpu-runtime.h]"]
  node_ssd_store_parser["Expert store parser<br/>[parser.cpp]"]
  node_ssd_stream["SSD streaming<br/>[runtime.cpp]"]
end

subgraph group_interfaces["User interfaces"]
  node_cli_chat["Terminal chat<br/>[chat.cpp]"]
  node_server["Inference server<br/>[server.cpp]"]
  node_web_chat["Browser chat<br/>[chat.service.ts]"]
end

subgraph group_grammar["Prompt and parsing"]
  node_jinja["Template runtime<br/>[parser.cpp]"]
  node_peg["PEG parser<br/>[peg-parser.cpp]"]
end

node_user(("User"))
node_api_client(("API clients"))

node_user -.->|"selects model"| node_launcher
node_launcher -.->|"checks capacity"| node_machine
node_launcher -.->|"resolves model"| node_model_source
node_launcher -.->|"downloads model"| node_model_store
node_model_store -.->|"provides weights"| node_model_loader
node_model_loader -.->|"builds architecture"| node_model_graphs
node_model_loader -.->|"configures runtime"| node_ssd_runtime
node_context -->|"runs SSD inference"| node_ssd_runtime
node_ssd_runtime -.->|"reads expert store"| node_ssd_store_parser
node_ssd_runtime -.->|"streams weights"| node_ssd_stream
node_context -.->|"builds compute graph"| node_tensor_ops
node_context -.->|"schedules execution"| node_backend
node_llama_api -.->|"exposes inference"| node_context
node_launcher -.->|"starts chat"| node_cli_chat
node_launcher -.->|"starts server"| node_server
node_server -.->|"serves inference"| node_context
node_api_client -.->|"sends requests"| node_server
node_user -.->|"chats in browser"| node_web_chat
node_web_chat -.->|"calls API"| node_server
node_server -.->|"applies templates"| node_jinja
node_common -.->|"supports parsing"| node_peg
node_common -.->|"supports templates"| node_jinja

click node_launcher "https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/app/command.py"
click node_machine "https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/machine/identity.py"
click node_model_source "https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/source/gguf_alias.py"
click node_model_store "https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/store/index.cpp"
click node_model_loader "https://github.com/wici-ai/inferssd/blob/main/engine/src/llama-context.cpp"
click node_model_graphs "https://github.com/wici-ai/inferssd/blob/main/engine/src/models/models.h"
click node_llama_api "https://github.com/wici-ai/inferssd/blob/main/engine/include/llama.h"
click node_context "https://github.com/wici-ai/inferssd/blob/main/engine/src/llama-context.cpp"
click node_tensor_ops "https://github.com/wici-ai/inferssd/blob/main/engine/ggml/include/ggml.h"
click node_backend "https://github.com/wici-ai/inferssd/blob/main/engine/ggml/include/ggml-backend.h"
click node_ssd_runtime "https://github.com/wici-ai/inferssd/blob/main/engine/src/ssd/cpu-runtime.h"
click node_ssd_store_parser "https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/store/parser.cpp"
click node_ssd_stream "https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/stream/runtime.cpp"
click node_cli_chat "https://github.com/wici-ai/inferssd/blob/main/engine/common/chat.cpp"
click node_server "https://github.com/wici-ai/inferssd/blob/main/engine/tools/server/server.cpp"
click node_web_chat "https://github.com/wici-ai/inferssd/blob/main/engine/tools/server/webui/src/lib/services/chat.service.ts"
click node_jinja "https://github.com/wici-ai/inferssd/blob/main/engine/common/jinja/parser.cpp"
click node_peg "https://github.com/wici-ai/inferssd/blob/main/engine/common/peg-parser.cpp"
click node_common "https://github.com/wici-ai/inferssd/blob/main/engine/common/common.h"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_launcher,node_machine,node_model_source,node_model_store,node_user,node_api_client toneBlue
class node_model_loader,node_model_graphs,node_llama_api,node_context,node_tensor_ops,node_backend,node_common toneAmber
class node_ssd_runtime,node_ssd_store_parser,node_ssd_stream toneMint
class node_cli_chat,node_server,node_web_chat toneRose
class node_jinja,node_peg toneIndigo
```

## Components

### Launch and models

- [CLI launcher](https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/app/command.py): `framework/src/inferssd/app/command.py`
- [Machine checks](https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/machine/identity.py): `framework/src/inferssd/machine/identity.py`
- [Model source](https://github.com/wici-ai/inferssd/blob/main/framework/src/inferssd/source/gguf_alias.py): `framework/src/inferssd/source/gguf_alias.py`
- [GGUF model store](https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/store/index.cpp): `engine/ggml/src/ggml-ssd/store/index.cpp`

### Inference engine

- [Model loading](https://github.com/wici-ai/inferssd/blob/main/engine/src/llama-context.cpp): `engine/src/llama-context.cpp`
- [Model architectures](https://github.com/wici-ai/inferssd/blob/main/engine/src/models/models.h): `engine/src/models/models.h`
- [Llama API](https://github.com/wici-ai/inferssd/blob/main/engine/include/llama.h): `engine/include/llama.h`
- [Inference context](https://github.com/wici-ai/inferssd/blob/main/engine/src/llama-context.cpp): `engine/src/llama-context.cpp`
- [Tensor operations](https://github.com/wici-ai/inferssd/blob/main/engine/ggml/include/ggml.h): `engine/ggml/include/ggml.h`
- [Compute backends](https://github.com/wici-ai/inferssd/blob/main/engine/ggml/include/ggml-backend.h): `engine/ggml/include/ggml-backend.h`
- [Shared inference support](https://github.com/wici-ai/inferssd/blob/main/engine/common/common.h): `engine/common/common.h`

### SSD execution

- [SSD runtime](https://github.com/wici-ai/inferssd/blob/main/engine/src/ssd/cpu-runtime.h): `engine/src/ssd/cpu-runtime.h`
- [Expert store parser](https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/store/parser.cpp): `engine/ggml/src/ggml-ssd/store/parser.cpp`
- [SSD streaming](https://github.com/wici-ai/inferssd/blob/main/engine/ggml/src/ggml-ssd/stream/runtime.cpp): `engine/ggml/src/ggml-ssd/stream/runtime.cpp`

### User interfaces

- [Terminal chat](https://github.com/wici-ai/inferssd/blob/main/engine/common/chat.cpp): `engine/common/chat.cpp`
- [Inference server](https://github.com/wici-ai/inferssd/blob/main/engine/tools/server/server.cpp): `engine/tools/server/server.cpp`
- [Browser chat](https://github.com/wici-ai/inferssd/blob/main/engine/tools/server/webui/src/lib/services/chat.service.ts): `engine/tools/server/webui/src/lib/services/chat.service.ts`

### Prompt and parsing

- [Template runtime](https://github.com/wici-ai/inferssd/blob/main/engine/common/jinja/parser.cpp): `engine/common/jinja/parser.cpp`
- [PEG parser](https://github.com/wici-ai/inferssd/blob/main/engine/common/peg-parser.cpp): `engine/common/peg-parser.cpp`

### Other components

- **User**
- **API clients**

## Connections

- User → CLI launcher: selects model (evidence: `README.md`)
- CLI launcher → Machine checks: checks capacity (evidence: `README.md`)
- CLI launcher → Model source: resolves model
- CLI launcher → GGUF model store: downloads model (evidence: `README.md`)
- GGUF model store → Model loading: provides weights (evidence: `README.md`)
- Model loading → Model architectures: builds architecture (evidence: `engine/src/llama-context.cpp`)
- Model loading → SSD runtime: configures runtime (evidence: `engine/src/llama-context.cpp`)
- Inference context → SSD runtime: runs SSD inference (evidence: `engine/src/llama-context.cpp`)
- SSD runtime → Expert store parser: reads expert store (evidence: `engine/src/ssd/cpu-runtime.h`)
- SSD runtime → SSD streaming: streams weights
- Inference context → Tensor operations: builds compute graph (evidence: `engine/src/llama-context.cpp`)
- Inference context → Compute backends: schedules execution (evidence: `engine/src/llama-context.cpp`)
- Llama API → Inference context: exposes inference (evidence: `engine/src/llama-context.cpp`)
- CLI launcher → Terminal chat: starts chat (evidence: `README.md`)
- CLI launcher → Inference server: starts server (evidence: `README.md`)
- Inference server → Inference context: serves inference (evidence: `README.md`)
- API clients → Inference server: sends requests (evidence: `README.md`)
- User → Browser chat: chats in browser (evidence: `README.md`)
- Browser chat → Inference server: calls API (evidence: `README.md`)
- Inference server → Template runtime: applies templates (evidence: `engine/tools/server/server.cpp`)
- Shared inference support → PEG parser: supports parsing (evidence: `engine/common/peg-parser.cpp`)
- Shared inference support → Template runtime: supports templates (evidence: `engine/common/jinja/parser.cpp`)