# ddxfish/sapphire architecture

> Architecture diagram of the GitHub repository ddxfish/sapphire, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ddxfish/sapphire
- Repository: https://github.com/ddxfish/sapphire

## Overview

1. **Top-level shape**
   - Sapphire is a **Python-backed, browser-based AI runtime** with a large **plugin ecosystem**.
   - The repo splits into a core backend (`main.py`, `sapphire.py`, `core/`), a web UI (`interfaces/web/`), and installable plugins (`plugins/`), with docs/tests around the edges.

2. **Entry points and deployment**
   - `main.py` and `sapphire.py` are the application launch paths.
   - `Dockerfile`, `docker-compose.yml`, and the `install/` requirements files define two main operating modes: local Python runtime and Dockerized web-only runtime.
   - `config.py` and the settings managers drive startup configuration, persona/provider selection, and feature gating.

3. **HTTP/API boundary**
   - `core/api_fastapi.py` and the route modules under `core/routes/` form the main server-side API surface.
   - The backend is organized by domain: chat, settings, system, media, knowledge, plugins, dashboard, store, TTS, videos, etc.
   - `core/api_tokens.py`, `core/auth.py`, and rate/permission logic control access to sensitive endpoints and integrations.

4. **Web UI boundary**
   - `interfaces/web/` is a substantial client-side app served by the backend via templates and static assets.
   - It is organized into shared components, feature modules, views, setup wizard flows, and API wrappers.
   - The UI consumes REST endpoints and streaming updates to render chat, settings, dashboards, personas, plugins, and voice controls.

5. **Chat and LLM orchestration**
   - `core/chat/` is the central conversational subsystem.
   - It handles prompt assembly, streaming responses, tool/function calling, chat history, formatting, and scope cleanup.
   - `core/chat/llm_providers/` abstracts model backends such as OpenAI-compatible endpoints, Anthropic/Claude, Gemini, and OpenAI Responses.
   - Prompt composition is managed separately via `core/prompts.py`, `core/prompt_manager.py`, `core/prompt_state.py`, and prompt defaults under `core/prompt_defaults/`.

6. **Persona, spice, and identity layer**
   - `core/personas/` and `core/spice_sets/` define the “who” and “style” of the assistant.
   - Personas bundle prompt fragments, tools, and provider choices; spice sets inject dynamic prompt variation.
   - This layer feeds directly into chat assembly so the same engine can behave as different characters or modes.

7. **Tool execution and plugin-mediated capabilities**
   - Tool selection and execution are handled by `core/toolsets/`, `core/tool_context.py`, and `core/chat/function_manager.py`.
   - Built-in tools live in `functions/`, while plugins can contribute additional tools, hooks, routes, widgets, and even providers.
   - `core/plugin_loader.py`, `core/plugin_verify.py`, and the plugin manifests/signatures enforce the plugin boundary and trust model.

8. **Voice pipeline**
   - Voice input/output is a dedicated subsystem: `core/wakeword/`, `core/stt/`, `core/tts/`, and `core/audio/`.
   - The flow is: wake word detection → audio capture → speech-to-text → chat → text-to-speech playback.
   - Provider adapters in STT/TTS let the system switch between local and remote engines, while the web UI can also participate in audio interaction.

9. **Memory, knowledge, and continuity**
   - Persistent cognition is split across `core/embeddings/`, `core/backup.py`, `core/migration.py`, `core/knowledge`-related routes, and the memory/goal/people/prompt managers.
   - `core/continuity/` provides autonomous scheduled execution contexts and schedulers for heartbeat-style tasks.
   - This is the long-lived state layer: memories, knowledge chunks, goals, people records, prompt state, and scheduled actions all feed back into chat behavior.

10. **Agents and background workers**
    - `core/agents/` implements background workers managed separately from the main chat loop.
    - Agents can run longer tasks, report progress later, and connect into the continuity/scheduler infrastructure.
    - This is distinct from immediate chat turns: agents are for asynchronous work, research, and multi-step jobs.

11. **Plugin architecture and extension surfaces**
    - Plugins are first-class packages under `plugins/` and can add:
      - tools,
      - routes,
      - daemons,
      - web apps,
      - themes,
      - widgets,
      - hooks,
      - providers,
      - voice commands.
    - Examples in the tree show integrations like Discord, Telegram, Email, Google Calendar, Home Assistant, GitHub, MCP, webcam, image generation, Bitcoin, SSH, and avatar support.
    - The architecture intentionally keeps these outside core so they can be installed, signed, verified, and removed independently.

12. **System/admin and operational layer**
    - `core/settings_manager.py`, `core/provider_registry.py`, `core/credentials_manager.py`, `core/process_manager.py`, `core/updater.py`, `core/metrics.py`, and related modules handle runtime administration.
    - `core/dashboard_builtins/` and `core/dashboard_widgets/` support the dashboard view and operational widgets.
    - `core/socks_proxy.py`, `core/ssl_utils.py`, `core/privacy.py`, and backup/migration code reflect the self-hosted, security-sensitive nature of the runtime.

13. **External services and boundaries**
    - Cloud LLMs, STT/TTS providers, email/calendar APIs, Discord/Telegram, Home Assistant, GitHub, ComfyUI, and MCP servers are all optional external boundaries.
    - The README’s privacy model is reflected in code: local-first is the default, and external connectivity is opt-in through providers/plugins/settings.

14. **How the main data flow fits together**
    - A user action starts in the web UI or voice subsystem.
    - The request enters FastAPI routes, gets authenticated/authorized, and is routed to chat, tools, agents, or plugin endpoints.
    - Chat composes prompts from persona/prompt/spice state, calls an LLM provider, optionally invokes tools, then streams output back to the UI and/or TTS.
    - Side effects may persist into history, memory, knowledge, goals, backups, plugin state, or scheduled continuations.

15. **What to emphasize in an architecture diagram**
    - Draw a central **core orchestration layer** between UI/voice and model/provider backends.
    - Show **plugins** as a parallel extension plane connected to routes, tools, hooks, daemons, and UI widgets.
    - Show **memory/knowledge/continuity** as persistent state services feeding the orchestration layer.
    - Show **voice** as a separate ingest/egress pipeline connected to chat.
    - Show **external integrations** only through explicit provider/plugin boundaries, not directly from the UI.

Main entry points: [`core/chat`](https://github.com/ddxfish/sapphire/tree/main/core/chat) (Chat core), [`plugins`](https://github.com/ddxfish/sapphire/tree/main/plugins) (Plugins), [`core/routes`](https://github.com/ddxfish/sapphire/tree/main/core/routes) (Routes), [`core/api_fastapi.py`](https://github.com/ddxfish/sapphire/blob/main/core/api_fastapi.py) (API server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_main["main.py<br/>app entry<br/>[main.py]"]
  node_sapphire_py["sapphire.py<br/>launcher<br/>[sapphire.py]"]
  node_config_py["config.py<br/>[config.py]"]
  node_api_fastapi["API server<br/>fastapi app<br/>[api_fastapi.py]"]
  node_routes["Routes<br/>api modules"]
  node_chat_core["Chat core<br/>orchestration"]
  node_llm_providers["LLM providers<br/>model adapters"]
  node_tooling["Tools<br/>tool runtime"]
end

subgraph group_ui["Web UI"]
  node_web_ui["Web UI<br/>browser app"]
  node_ui_templates["Templates<br/>html shell"]
  node_ui_static["Static app<br/>frontend modules"]
end

subgraph group_state["State"]
  node_prompts["Prompts<br/>prompt system<br/>[prompts.py]"]
  node_personas["Personas<br/>identity layer"]
  node_spice_sets["Spices<br/>prompt variation"]
  node_memory[("Memory<br/>knowledge state")]
  node_continuity["Continuity<br/>scheduler"]
end

subgraph group_voice["Voice"]
  node_voice_pipeline["Voice pipeline<br/>audio pipeline"]
end

subgraph group_extensions["Extensions"]
  node_plugin_loader["Plugin loader<br/>plugin boundary<br/>[plugin_loader.py]"]
  node_plugins_root["Plugins<br/>extension packages"]
end

subgraph group_ops["Ops"]
  node_ops_core["Ops core<br/>admin services"]
  node_dashboard["Dashboard<br/>ops ui"]
end

node_main -->|"loads"| node_config_py
node_sapphire_py -->|"starts"| node_api_fastapi
node_main -->|"starts"| node_api_fastapi
node_api_fastapi -->|"mounts"| node_routes
node_routes -->|"dispatches"| node_chat_core
node_routes -->|"exposes"| node_voice_pipeline
node_chat_core -->|"calls"| node_llm_providers
node_chat_core -->|"uses"| node_tooling
node_chat_core -->|"assembles"| node_prompts
node_chat_core -->|"selects"| node_personas
node_chat_core -->|"varies"| node_spice_sets
node_chat_core -->|"reads/writes"| node_memory
node_chat_core -->|"schedules"| node_continuity
node_voice_pipeline -->|"feeds"| node_chat_core
node_chat_core -->|"speaks to"| node_voice_pipeline
node_web_ui -->|"renders"| node_ui_templates
node_web_ui -->|"serves"| node_ui_static
node_ui_static -->|"calls"| node_routes
node_plugin_loader -->|"loads"| node_plugins_root
node_plugins_root -->|"extends"| node_tooling
node_plugins_root -->|"extends"| node_routes
node_plugins_root -->|"extends"| node_web_ui
node_plugins_root -->|"extends"| node_voice_pipeline
node_plugins_root -->|"integrates"| node_ops_core
node_ops_core -->|"drives"| node_dashboard
node_ops_core -->|"supports"| node_api_fastapi

click node_main "https://github.com/ddxfish/sapphire/blob/main/main.py"
click node_sapphire_py "https://github.com/ddxfish/sapphire/blob/main/sapphire.py"
click node_config_py "https://github.com/ddxfish/sapphire/blob/main/config.py"
click node_api_fastapi "https://github.com/ddxfish/sapphire/blob/main/core/api_fastapi.py"
click node_routes "https://github.com/ddxfish/sapphire/tree/main/core/routes"
click node_chat_core "https://github.com/ddxfish/sapphire/tree/main/core/chat"
click node_llm_providers "https://github.com/ddxfish/sapphire/tree/main/core/chat/llm_providers"
click node_tooling "https://github.com/ddxfish/sapphire/tree/main/core/toolsets"
click node_prompts "https://github.com/ddxfish/sapphire/blob/main/core/prompts.py"
click node_personas "https://github.com/ddxfish/sapphire/tree/main/core/personas"
click node_spice_sets "https://github.com/ddxfish/sapphire/tree/main/core/spice_sets"
click node_memory "https://github.com/ddxfish/sapphire/tree/main/core/embeddings"
click node_continuity "https://github.com/ddxfish/sapphire/tree/main/core/continuity"
click node_voice_pipeline "https://github.com/ddxfish/sapphire/tree/main/core/wakeword"
click node_web_ui "https://github.com/ddxfish/sapphire/tree/main/interfaces/web"
click node_ui_templates "https://github.com/ddxfish/sapphire/tree/main/interfaces/web/templates"
click node_ui_static "https://github.com/ddxfish/sapphire/tree/main/interfaces/web/static"
click node_plugin_loader "https://github.com/ddxfish/sapphire/blob/main/core/plugin_loader.py"
click node_plugins_root "https://github.com/ddxfish/sapphire/tree/main/plugins"
click node_ops_core "https://github.com/ddxfish/sapphire/blob/main/core/settings_manager.py"
click node_dashboard "https://github.com/ddxfish/sapphire/tree/main/core/dashboard_builtins"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_sapphire_py,node_config_py,node_api_fastapi,node_routes,node_chat_core,node_llm_providers,node_tooling toneBlue
class node_web_ui,node_ui_templates,node_ui_static toneAmber
class node_prompts,node_personas,node_spice_sets,node_memory,node_continuity toneMint
class node_voice_pipeline toneRose
class node_plugin_loader,node_plugins_root toneIndigo
class node_ops_core,node_dashboard toneTeal
```

## Components

### Runtime

Launch, config, server, and core orchestration

- [main.py](https://github.com/ddxfish/sapphire/blob/main/main.py): app entry · Primary process entrypoint. · `main.py`
- [sapphire.py](https://github.com/ddxfish/sapphire/blob/main/sapphire.py): launcher · Secondary launch path for the runtime. · `sapphire.py`
- [config.py](https://github.com/ddxfish/sapphire/blob/main/config.py): config · Startup configuration and feature gating. · `config.py`
- [API server](https://github.com/ddxfish/sapphire/blob/main/core/api_fastapi.py): fastapi app · Main HTTP boundary and request dispatcher. · `core/api_fastapi.py`
- [Routes](https://github.com/ddxfish/sapphire/tree/main/core/routes): api modules · Domain route layer for chat, settings, system, media, knowledge, plugins, dashboard, store, TTS, and videos. · `core/routes`
- [Chat core](https://github.com/ddxfish/sapphire/tree/main/core/chat): orchestration · Prompt assembly, streaming, tool calling, history, formatting, and cleanup. · `core/chat`
- [LLM providers](https://github.com/ddxfish/sapphire/tree/main/core/chat/llm_providers): model adapters · Provider backends such as OpenAI-compatible, Anthropic, Gemini, and Responses. · `core/chat/llm_providers`
- [Tools](https://github.com/ddxfish/sapphire/tree/main/core/toolsets): tool runtime · Tool context, toolsets, and function execution for chat and plugins. · `core/toolsets`

### Web UI

Browser client and templates served by the backend

- [Web UI](https://github.com/ddxfish/sapphire/tree/main/interfaces/web): browser app · Browser-based client served through templates and static assets. · `interfaces/web`
- [Templates](https://github.com/ddxfish/sapphire/tree/main/interfaces/web/templates): html shell · HTML entry templates for app, login, and setup. · `interfaces/web/templates`
- [Static app](https://github.com/ddxfish/sapphire/tree/main/interfaces/web/static): frontend modules · Client runtime, shared components, views, and streaming wrappers. · `interfaces/web/static`

### State

Persistent memory, prompts, personas, settings, and embeddings

- [Prompts](https://github.com/ddxfish/sapphire/blob/main/core/prompts.py): prompt system · Prompt defaults, state, CRUD, and composition utilities. · `core/prompts.py`
- [Personas](https://github.com/ddxfish/sapphire/tree/main/core/personas): identity layer · Persona definitions and persona manager. · `core/personas`
- [Spices](https://github.com/ddxfish/sapphire/tree/main/core/spice_sets): prompt variation · Dynamic style and prompt variation sets. · `core/spice_sets`
- [Memory](https://github.com/ddxfish/sapphire/tree/main/core/embeddings): knowledge state · Embeddings, knowledge, backup, migration, and continuity state. · `core/embeddings`
- [Continuity](https://github.com/ddxfish/sapphire/tree/main/core/continuity): scheduler · Scheduled execution contexts and autonomous task scheduling. · `core/continuity`

### Voice

Wakeword, STT, TTS, and audio handling

- [Voice pipeline](https://github.com/ddxfish/sapphire/tree/main/core/wakeword): audio pipeline · Wake word, recording, STT, and TTS subsystems. · `core/wakeword`

### Extensions

Plugin loading, verification, tools, routes, daemons, and widgets

- [Plugin loader](https://github.com/ddxfish/sapphire/blob/main/core/plugin_loader.py): plugin boundary · Plugin loading, verification, and trust enforcement. · `core/plugin_loader.py`
- [Plugins](https://github.com/ddxfish/sapphire/tree/main/plugins): extension packages · Installable plugins contributing tools, routes, daemons, widgets, hooks, and providers. · `plugins`

### Ops

Administration, metrics, updates, and system controls

- [Ops core](https://github.com/ddxfish/sapphire/blob/main/core/settings_manager.py): admin services · Settings, provider registry, credentials, processes, updates, metrics, privacy, and proxy/security helpers. · `core/settings_manager.py`
- [Dashboard](https://github.com/ddxfish/sapphire/tree/main/core/dashboard_builtins): ops ui · Built-in dashboard widgets and operational screens. · `core/dashboard_builtins`

## Connections

- main.py → config.py: loads · Bootstraps runtime configuration.
- sapphire.py → API server: starts · Launches the server application.
- main.py → API server: starts · Primary entry into the HTTP server.
- API server → Routes: mounts · Registers domain API routes.
- Routes → Chat core: dispatches · Chat endpoints flow into orchestration.
- Routes → Voice pipeline: exposes · Audio and voice endpoints connect to the voice pipeline.
- Chat core → LLM providers: calls · Chat delegates model execution to provider adapters.
- Chat core → Tools: uses · Chat invokes tools and tool context when needed.
- Chat core → Prompts: assembles · Prompt composition depends on prompt state and defaults.
- Chat core → Personas: selects · Persona choice shapes assistant behavior.
- Chat core → Spices: varies · Spice sets inject style and variation.
- Chat core → Memory: reads/writes · Chat persists history, memory, and knowledge.
- Chat core → Continuity: schedules · Long-running tasks and scheduled follow-ups feed continuity.
- Voice pipeline → Chat core: feeds · Transcribed speech enters chat orchestration.
- Chat core → Voice pipeline: speaks to · Responses can be rendered back through TTS.
- Web UI → Templates: renders · Templates provide the HTML shell.
- Web UI → Static app: serves · Static client modules implement the app.
- Static app → Routes: calls · Frontend consumes REST and streaming endpoints.
- Plugin loader → Plugins: loads · Discovers and verifies installed plugins.
- Plugins → Tools: extends · Plugins contribute tools to the runtime.
- Plugins → Routes: extends · Plugins can add API routes.
- Plugins → Web UI: extends · Plugins can add widgets and frontend modules.
- Plugins → Voice pipeline: extends · Plugins can add voice commands and audio behavior.
- Plugins → Ops core: integrates · Plugins can expose settings and integrations.
- Ops core → Dashboard: drives · Operational services power the dashboard.
- Ops core → API server: supports · Admin and runtime controls back the server boundary.