wici-ai/inferssd

How wici-ai/inferssd works

InferSSD runs large GGUF mixture-of-experts models by placing expert weights on SSD while performing inference through its llama.cpp-derived engine. The user selects a model and starts CLI chat or the server; model acquisition and machine checks precede loading, then inference produces generated tokens for the chosen interface. The repository also includes a browser chat UI, OpenAI-compatible endpoints, and ordinary RAM inference for Gemma. Several launcher and storage internals are unsampled, so those relationships are shown only where documented or explicitly marked unverified.

Main entry points: command.py CLI launcher · llama-context.cpp Inference context · server.cpp Inference server · cpu-runtime.h SSD runtime

Components 21

Launch and models

CLI launcher framework/src/inferssd/app/command.py
Machine checks framework/src/inferssd/machine/identity.py
Model source framework/src/inferssd/source/gguf_alias.py
GGUF model store engine/ggml/src/ggml-ssd/store/index.cpp

Inference engine

Model loading engine/src/llama-context.cpp
Model architectures engine/src/models/models.h
Llama API engine/include/llama.h
Inference context engine/src/llama-context.cpp
Tensor operations engine/ggml/include/ggml.h
Compute backends engine/ggml/include/ggml-backend.h
Shared inference support engine/common/common.h

SSD execution

SSD runtime engine/src/ssd/cpu-runtime.h
Expert store parser engine/ggml/src/ggml-ssd/store/parser.cpp
SSD streaming engine/ggml/src/ggml-ssd/stream/runtime.cpp

User interfaces

Terminal chat engine/common/chat.cpp
Inference server engine/tools/server/server.cpp
Browser chat engine/tools/server/webui/src/lib/services/chat.service.ts

Prompt and parsing

Template runtime engine/common/jinja/parser.cpp
PEG parser engine/common/peg-parser.cpp

Other components

User
API clients

Connections 22

User → CLI launcher: selects model README.md
CLI launcher → Machine checks: checks capacity README.md
CLI launcher → Model source: resolves model
CLI launcher → GGUF model store: downloads model README.md
GGUF model store → Model loading: provides weights README.md
Model loading → Model architectures: builds architecture engine/src/llama-context.cpp
Model loading → SSD runtime: configures runtime engine/src/llama-context.cpp
Inference context → SSD runtime: runs SSD inference engine/src/llama-context.cpp
SSD runtime → Expert store parser: reads expert store engine/src/ssd/cpu-runtime.h
SSD runtime → SSD streaming: streams weights
Inference context → Tensor operations: builds compute graph engine/src/llama-context.cpp
Inference context → Compute backends: schedules execution engine/src/llama-context.cpp
Llama API → Inference context: exposes inference engine/src/llama-context.cpp
CLI launcher → Terminal chat: starts chat README.md
CLI launcher → Inference server: starts server README.md
Inference server → Inference context: serves inference README.md
API clients → Inference server: sends requests README.md
User → Browser chat: chats in browser README.md
Browser chat → Inference server: calls API README.md
Inference server → Template runtime: applies templates engine/tools/server/server.cpp
Shared inference support → PEG parser: supports parsing engine/common/peg-parser.cpp
Shared inference support → Template runtime: supports templates engine/common/jinja/parser.cpp

Drawn by GitDiagram on November 14, 2026 from the repository's file tree, README and source files. Read as Markdown ·