How ddxfish/sapphire works
1. Top-level shape
- Sapphire is a Python-backed, browser-based AI runtime with a large plugin ecosystem.
- The repo splits into a core backend (main.py, sapphire.py, core/), a web UI (interfaces/web/), and installable plugins (plugins/), with docs/tests around the edges.
2. Entry points and deployment
- main.py and sapphire.py are the application launch paths.
- Dockerfile, docker-compose.yml, and the install/ requirements files define two main operating modes: local Python runtime and Dockerized web-only runtime.
- config.py and the settings managers drive startup configuration, persona/provider selection, and feature gating.
3. HTTP/API boundary
- core/api_fastapi.py and the route modules under core/routes/ form the main server-side API surface.
- The backend is organized by domain: chat, settings, system, media, knowledge, plugins, dashboard, store, TTS, videos, etc.
- core/api_tokens.py, core/auth.py, and rate/permission logic control access to sensitive endpoints and integrations.
4. Web UI boundary
- interfaces/web/ is a substantial client-side app served by the backend via templates and static assets.
- It is organized into shared components, feature modules, views, setup wizard flows, and API wrappers.
- The UI consumes REST endpoints and streaming updates to render chat, settings, dashboards, personas, plugins, and voice controls.
5. Chat and LLM orchestration
- core/chat/ is the central conversational subsystem.
- It handles prompt assembly, streaming responses, tool/function calling, chat history, formatting, and scope cleanup.
- core/chat/llm_providers/ abstracts model backends such as OpenAI-compatible endpoints, Anthropic/Claude, Gemini, and OpenAI Responses.
- Prompt composition is managed separately via core/prompts.py, core/prompt_manager.py, core/prompt_state.py, and prompt defaults under core/prompt_defaults/.
6. Persona, spice, and identity layer
- core/personas/ and core/spice_sets/ define the “who” and “style” of the assistant.
- Personas bundle prompt fragments, tools, and provider choices; spice sets inject dynamic prompt variation.
- This layer feeds directly into chat assembly so the same engine can behave as different characters or modes.
7. Tool execution and plugin-mediated capabilities
- Tool selection and execution are handled by core/toolsets/, core/tool_context.py, and core/chat/function_manager.py.
- Built-in tools live in functions/, while plugins can contribute additional tools, hooks, routes, widgets, and even providers.
- core/plugin_loader.py, core/plugin_verify.py, and the plugin manifests/signatures enforce the plugin boundary and trust model.
8. Voice pipeline
- Voice input/output is a dedicated subsystem: core/wakeword/, core/stt/, core/tts/, and core/audio/.
- The flow is: wake word detection → audio capture → speech-to-text → chat → text-to-speech playback.
- Provider adapters in STT/TTS let the system switch between local and remote engines, while the web UI can also participate in audio interaction.
9. Memory, knowledge, and continuity
- Persistent cognition is split across core/embeddings/, core/backup.py, core/migration.py, core/knowledge-related routes, and the memory/goal/people/prompt managers.
- core/continuity/ provides autonomous scheduled execution contexts and schedulers for heartbeat-style tasks.
- This is the long-lived state layer: memories, knowledge chunks, goals, people records, prompt state, and scheduled actions all feed back into chat behavior.
10. Agents and background workers
- core/agents/ implements background workers managed separately from the main chat loop.
- Agents can run longer tasks, report progress later, and connect into the continuity/scheduler infrastructure.
- This is distinct from immediate chat turns: agents are for asynchronous work, research, and multi-step jobs.
11. Plugin architecture and extension surfaces
- Plugins are first-class packages under plugins/ and can add:
- tools,
- routes,
- daemons,
- web apps,
- themes,
- widgets,
- hooks,
- providers,
- voice commands.
- Examples in the tree show integrations like Discord, Telegram, Email, Google Calendar, Home Assistant, GitHub, MCP, webcam, image generation, Bitcoin, SSH, and avatar support.
- The architecture intentionally keeps these outside core so they can be installed, signed, verified, and removed independently.
12. System/admin and operational layer
- core/settings_manager.py, core/provider_registry.py, core/credentials_manager.py, core/process_manager.py, core/updater.py, core/metrics.py, and related modules handle runtime administration.
- core/dashboard_builtins/ and core/dashboard_widgets/ support the dashboard view and operational widgets.
- core/socks_proxy.py, core/ssl_utils.py, core/privacy.py, and backup/migration code reflect the self-hosted, security-sensitive nature of the runtime.
13. External services and boundaries
- Cloud LLMs, STT/TTS providers, email/calendar APIs, Discord/Telegram, Home Assistant, GitHub, ComfyUI, and MCP servers are all optional external boundaries.
- The README’s privacy model is reflected in code: local-first is the default, and external connectivity is opt-in through providers/plugins/settings.
14. How the main data flow fits together
- A user action starts in the web UI or voice subsystem.
- The request enters FastAPI routes, gets authenticated/authorized, and is routed to chat, tools, agents, or plugin endpoints.
- Chat composes prompts from persona/prompt/spice state, calls an LLM provider, optionally invokes tools, then streams output back to the UI and/or TTS.
- Side effects may persist into history, memory, knowledge, goals, backups, plugin state, or scheduled continuations.
15. What to emphasize in an architecture diagram
- Draw a central core orchestration layer between UI/voice and model/provider backends.
- Show plugins as a parallel extension plane connected to routes, tools, hooks, daemons, and UI widgets.
- Show memory/knowledge/continuity as persistent state services feeding the orchestration layer.
- Show voice as a separate ingest/egress pipeline connected to chat.
- Show external integrations only through explicit provider/plugin boundaries, not directly from the UI.
Main entry points: chat Chat core · plugins Plugins · routes Routes · api_fastapi.py API server
Components 21
Runtime
Launch, config, server, and core orchestration
- main.py · Primary process entrypoint.
main.py - sapphire.py · Secondary launch path for the runtime.
sapphire.py - config.py · Startup configuration and feature gating.
config.py - API server · Main HTTP boundary and request dispatcher.
core/api_fastapi.py - Routes · Domain route layer for chat, settings, system, media, knowledge, plugins, dashboard, store, TTS, and videos.
core/routes - Chat core · Prompt assembly, streaming, tool calling, history, formatting, and cleanup.
core/chat - LLM providers · Provider backends such as OpenAI-compatible, Anthropic, Gemini, and Responses.
core/chat/llm_providers - Tools · Tool context, toolsets, and function execution for chat and plugins.
core/toolsets
Web UI
Browser client and templates served by the backend
- Web UI · Browser-based client served through templates and static assets.
interfaces/web - Templates · HTML entry templates for app, login, and setup.
interfaces/web/templates - Static app · Client runtime, shared components, views, and streaming wrappers.
interfaces/web/static
State
Persistent memory, prompts, personas, settings, and embeddings
- Prompts · Prompt defaults, state, CRUD, and composition utilities.
core/prompts.py - Personas · Persona definitions and persona manager.
core/personas - Spices · Dynamic style and prompt variation sets.
core/spice_sets - Memory · Embeddings, knowledge, backup, migration, and continuity state.
core/embeddings - Continuity · Scheduled execution contexts and autonomous task scheduling.
core/continuity
Voice
Wakeword, STT, TTS, and audio handling
- Voice pipeline · Wake word, recording, STT, and TTS subsystems.
core/wakeword
Extensions
Plugin loading, verification, tools, routes, daemons, and widgets
- Plugin loader · Plugin loading, verification, and trust enforcement.
core/plugin_loader.py - Plugins · Installable plugins contributing tools, routes, daemons, widgets, hooks, and providers.
plugins
Connections 26
- main.py → config.py: loads · Bootstraps runtime configuration.
- sapphire.py → API server: starts · Launches the server application.
- main.py → API server: starts · Primary entry into the HTTP server.
- API server → Routes: mounts · Registers domain API routes.
- Routes → Chat core: dispatches · Chat endpoints flow into orchestration.
- Routes → Voice pipeline: exposes · Audio and voice endpoints connect to the voice pipeline.
- Chat core → LLM providers: calls · Chat delegates model execution to provider adapters.
- Chat core → Tools: uses · Chat invokes tools and tool context when needed.
- Chat core → Prompts: assembles · Prompt composition depends on prompt state and defaults.
- Chat core → Personas: selects · Persona choice shapes assistant behavior.
- Chat core → Spices: varies · Spice sets inject style and variation.
- Chat core → Memory: reads/writes · Chat persists history, memory, and knowledge.
- Chat core → Continuity: schedules · Long-running tasks and scheduled follow-ups feed continuity.
- Voice pipeline → Chat core: feeds · Transcribed speech enters chat orchestration.
- Chat core → Voice pipeline: speaks to · Responses can be rendered back through TTS.
- Web UI → Templates: renders · Templates provide the HTML shell.
- Web UI → Static app: serves · Static client modules implement the app.
- Static app → Routes: calls · Frontend consumes REST and streaming endpoints.
- Plugin loader → Plugins: loads · Discovers and verifies installed plugins.
- Plugins → Tools: extends · Plugins contribute tools to the runtime.
- Plugins → Routes: extends · Plugins can add API routes.
- Plugins → Web UI: extends · Plugins can add widgets and frontend modules.
- Plugins → Voice pipeline: extends · Plugins can add voice commands and audio behavior.
- Plugins → Ops core: integrates · Plugins can expose settings and integrations.
- Ops core → Dashboard: drives · Operational services power the dashboard.
- Ops core → API server: supports · Admin and runtime controls back the server boundary.