# braedynl0530/desktoppetml architecture

> Architecture diagram of the GitHub repository braedynl0530/desktoppetml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/braedynl0530/desktoppetml
- Repository: https://github.com/braedynl0530/desktoppetml

## Overview

1. **Overall shape**
   This repo is a local, event-driven desktop companion with two execution surfaces: a **desktop pet UI** and a **Minecraft fake-player bot**. Both are coordinated by shared “core” services rather than by a single monolith.

2. **Primary entry points**
   - `ui/pet.py` is the main launcher for the desktop experience and auto-selects GUI vs TUI.
   - `ui/tui.py` is a standalone terminal mode.
   - `core/local_agent_server.py` exposes a small local HTTP service for planning/summarization tasks.
   - `minecraft/petbot_main.sc` is the Scarpet script entry point on the Minecraft side.

3. **Core runtime and configuration**
   `core/config.py` centralizes feature flags, polling rates, model/provider selection, memory limits, and UI mode. The app is highly environment-variable driven, which means architecture and behavior change without code changes. `core/platform_utils.py` likely abstracts OS-specific behavior such as window detection and process checks.

4. **Desktop sensing and behavior loop**
   The desktop pet monitors user activity through `core/tracking.py`, with memory and state handled by `core/memory.py`, `core/short_memory.py`, and `core/Tiered_memory.py`. `core/item_preferences.py` and `core/personalityEngine.py` turn those observations into mood, preference, and dialog decisions. This is the main feedback loop: observe apps/events → classify → store → infer behavior → react.

5. **Speech and messaging**
   `core/STT.py` handles speech-to-text input, while `core/messaging.py` appears to route user-facing text/chat bubble messages. These inputs are normalized before reaching the LLM or command system, with debounce and retry controls coming from config.

6. **LLM boundary**
   The `llm/` package is the model-facing layer:
   - `llm/ollama_client.py` is the local model client.
   - `llm/intents.py` defines intent classification or command mapping.
   - `llm/response_parser.py` turns model output into structured actions.
   - `llm/prompts/` contains system prompts for personality, reasoning, and Minecraft behavior.
   
   Architecturally, this is the boundary between “natural language” and “actionable commands.”

7. **Agent orchestration**
   `core/agents.py` and `core/agent_bridge.py` are the coordination layer that converts tracked state, STT, and LLM output into concrete actions. `core/mcp_client.py` and the MCP-related settings in config indicate an integration path for external tools/services, especially Obsidian. `core/local_agent_service.py` likely provides a small local service interface for note/planning outputs.

8. **Plugin system**
   `core/plugin_system.py` loads optional plugins from `core/plugins/`. The visible plugins are:
   - `obsidian_plugin.py` for Obsidian/MCP note workflows.
   - `tui_plugin.py` for terminal interaction support.
   
   Plugins are a clear extension boundary: they consume core events/commands without owning the main loop.

9. **Minecraft subsystem**
   `minecraft/minecraft_agent.py` and `minecraft/minecraft_bridge.py` form a separate but connected behavior stack for the in-game fake player. The README indicates communication via an HTTP bridge, with the Minecraft side sending context such as chat, position, held items, and nearby blocks, and receiving actions back. This subsystem has its own personality, memory, and action vocabulary, but still depends on the shared LLM and core decision logic.

10. **UI and presentation layer**
   `ui/pet.py` contains the GUI desktop pet, likely using PyQt5 for animations, chat bubbles, and mic controls. `ui/tui.py` provides the terminal UI and command entry path. The presentation layer is intentionally thin: it displays state and forwards user interaction into the core/LLM pipeline.

11. **Data persistence and learning**
   The repository uses **SQLite** as the durable store for behavior history and model/tracking data. The learning path described in the README is: application/window events → persisted records → anomaly detection / habit modeling → personality and reaction adjustments. `core/tracking.py` and the memory modules are the main persistence-to-inference bridge.

12. **External dependencies and system integration**
   Several outside systems materially affect the architecture:
   - **Ollama** for local inference, with optional **Gemini** provider support.
   - **Fabric/Carpet** and Scarpet for Minecraft automation.
   - **psutil** for Minecraft process detection.
   - **xdotool** or platform APIs for active-window detection.
   - Optional **Obsidian MCP** transport for note operations.
   - `requirements.txt` and `installer stuff/hook-all.py` support packaging/deployment.

13. **Main data flows to diagram**
   - **Desktop flow:** user activity/STT → tracking/memory → LLM/intents → personality/messaging → UI animations/chat.
   - **Minecraft flow:** game state/chat → bridge/agent → LLM/prompting → response parser → in-game action.
   - **Planning/notes flow:** typed/STT command → plugin/MCP or local agent service → summarized/structured output.
   
   These flows share the same core philosophy: event capture, stateful memory, LLM interpretation, then domain-specific action.

Main entry points: [`core/agents.py`](https://github.com/braedynl0530/desktoppetml/blob/master/core/agents.py) (Agents), [`llm/ollama_client.py`](https://github.com/braedynl0530/desktoppetml/blob/master/llm/ollama_client.py) (Ollama client), [`minecraft/minecraft_bridge.py`](https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/minecraft_bridge.py) (Minecraft bridge), [`core/config.py`](https://github.com/braedynl0530/desktoppetml/blob/master/core/config.py) (Config)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_core["Core runtime"]
  node_n_core_config["Config<br/>settings/runtime<br/>[config.py]"]
  node_n_core_tracking["Tracking<br/>sensing<br/>[tracking.py]"]
  node_n_core_memory[("Memory<br/>state/store<br/>[memory.py]")]
  node_n_core_short_memory["Short memory<br/>ephemeral state<br/>[short_memory.py]"]
  node_n_core_tiered_memory[("Tiered memory<br/>memory policy<br/>[Tiered_memory.py]")]
  node_n_core_personality["Personality<br/>behavior engine"]
  node_n_core_preferences["Preferences<br/>habit model"]
  node_n_core_stt["Speech input<br/>stt<br/>[STT.py]"]
  node_n_core_messaging["Messaging<br/>chat/output<br/>[messaging.py]"]
  node_n_core_agents["Agents<br/>orchestration<br/>[agents.py]"]
  node_n_core_bridge["Agent bridge<br/>command bridge<br/>[agent_bridge.py]"]
  node_n_core_service["Local service<br/>local api"]
  node_n_core_server["Local server<br/>local api"]
  node_n_core_mcp["MCP client<br/>tool bridge<br/>[mcp_client.py]"]
  node_n_core_plugins["Plugin system<br/>extensions<br/>[plugin_system.py]"]
end

subgraph group_group_llm["LLM layer"]
  node_n_llm_client["Ollama client<br/>llm client<br/>[ollama_client.py]"]
  node_n_llm_intents["Intents<br/>intent map<br/>[intents.py]"]
  node_n_llm_parser["Response parser<br/>structured output<br/>[response_parser.py]"]
  node_n_llm_prompts["Prompts<br/>prompt assets"]
end

subgraph group_group_ui["Desktop UI"]
  node_n_ui_pet["Desktop pet<br/>launcher/ui<br/>[pet.py]"]
  node_n_ui_tui["Terminal UI<br/>tui/ui<br/>[tui.py]"]
end

subgraph group_group_minecraft["Minecraft bot"]
  node_n_mc_entry["Petbot main<br/>scarpet entry<br/>[petbot_main.sc]"]
  node_n_mc_agent["Minecraft agent<br/>game agent<br/>[minecraft_agent.py]"]
  node_n_mc_bridge["Minecraft bridge<br/>http bridge"]
end

subgraph group_group_plugins["Plugins"]
  node_n_plugin_obsidian["Obsidian plugin<br/>[obsidian_plugin.py]"]
  node_n_plugin_tui["TUI plugin<br/>[tui_plugin.py]"]
end

node_n_ui_pet -->|"reads mode"| node_n_core_config
node_n_ui_pet -->|"drives"| node_n_core_agents
node_n_ui_tui -->|"sends input"| node_n_core_agents
node_n_core_tracking -->|"persists"| node_n_core_memory
node_n_core_tracking -->|"updates"| node_n_core_short_memory
node_n_core_short_memory -->|"rolls up"| node_n_core_tiered_memory
node_n_core_memory -->|"feeds"| node_n_core_personality
node_n_core_preferences -->|"shapes"| node_n_core_personality
node_n_core_stt -->|"input"| node_n_core_agents
node_n_core_agents -->|"classifies"| node_n_llm_intents
node_n_core_agents -->|"queries"| node_n_llm_client
node_n_llm_client -->|"uses"| node_n_llm_prompts
node_n_llm_client -->|"returns text"| node_n_llm_parser
node_n_llm_parser -->|"emits actions"| node_n_core_bridge
node_n_core_bridge -->|"routes"| node_n_core_messaging
node_n_core_bridge -->|"dispatches"| node_n_core_plugins
node_n_core_plugins -->|"loads"| node_n_plugin_obsidian
node_n_core_plugins -->|"loads"| node_n_plugin_tui
node_n_core_service -->|"serves via"| node_n_core_server
node_n_core_service -->|"supports"| node_n_core_agents
node_n_core_mcp -->|"connects"| node_n_plugin_obsidian
node_n_mc_entry -->|"starts"| node_n_mc_bridge
node_n_mc_bridge -->|"coordinates"| node_n_mc_agent
node_n_mc_bridge -->|"calls"| node_n_llm_client
node_n_mc_bridge -->|"uses"| node_n_llm_prompts
node_n_mc_agent -->|"interprets"| node_n_llm_parser
node_n_mc_agent -.->|"shares state"| node_n_core_memory
node_n_core_config -->|"tunes"| node_n_core_tracking
node_n_core_config -->|"selects provider"| node_n_llm_client

click node_n_ui_pet "https://github.com/braedynl0530/desktoppetml/blob/master/ui/pet.py"
click node_n_ui_tui "https://github.com/braedynl0530/desktoppetml/blob/master/ui/tui.py"
click node_n_core_config "https://github.com/braedynl0530/desktoppetml/blob/master/core/config.py"
click node_n_core_tracking "https://github.com/braedynl0530/desktoppetml/blob/master/core/tracking.py"
click node_n_core_memory "https://github.com/braedynl0530/desktoppetml/blob/master/core/memory.py"
click node_n_core_short_memory "https://github.com/braedynl0530/desktoppetml/blob/master/core/short_memory.py"
click node_n_core_tiered_memory "https://github.com/braedynl0530/desktoppetml/blob/master/core/Tiered_memory.py"
click node_n_core_personality "https://github.com/braedynl0530/desktoppetml/blob/master/core/personalityEngine.py"
click node_n_core_preferences "https://github.com/braedynl0530/desktoppetml/blob/master/core/item_preferences.py"
click node_n_core_stt "https://github.com/braedynl0530/desktoppetml/blob/master/core/STT.py"
click node_n_core_messaging "https://github.com/braedynl0530/desktoppetml/blob/master/core/messaging.py"
click node_n_core_agents "https://github.com/braedynl0530/desktoppetml/blob/master/core/agents.py"
click node_n_core_bridge "https://github.com/braedynl0530/desktoppetml/blob/master/core/agent_bridge.py"
click node_n_core_service "https://github.com/braedynl0530/desktoppetml/blob/master/core/local_agent_service.py"
click node_n_core_server "https://github.com/braedynl0530/desktoppetml/blob/master/core/local_agent_server.py"
click node_n_core_mcp "https://github.com/braedynl0530/desktoppetml/blob/master/core/mcp_client.py"
click node_n_core_plugins "https://github.com/braedynl0530/desktoppetml/blob/master/core/plugin_system.py"
click node_n_plugin_obsidian "https://github.com/braedynl0530/desktoppetml/blob/master/core/plugins/obsidian_plugin.py"
click node_n_plugin_tui "https://github.com/braedynl0530/desktoppetml/blob/master/core/plugins/tui_plugin.py"
click node_n_llm_client "https://github.com/braedynl0530/desktoppetml/blob/master/llm/ollama_client.py"
click node_n_llm_intents "https://github.com/braedynl0530/desktoppetml/blob/master/llm/intents.py"
click node_n_llm_parser "https://github.com/braedynl0530/desktoppetml/blob/master/llm/response_parser.py"
click node_n_llm_prompts "https://github.com/braedynl0530/desktoppetml/tree/master/llm/prompts"
click node_n_mc_entry "https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/petbot_main.sc"
click node_n_mc_agent "https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/minecraft_agent.py"
click node_n_mc_bridge "https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/minecraft_bridge.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_core_config,node_n_core_tracking,node_n_core_memory,node_n_core_short_memory,node_n_core_tiered_memory,node_n_core_personality,node_n_core_preferences,node_n_core_stt,node_n_core_messaging,node_n_core_agents,node_n_core_bridge,node_n_core_service,node_n_core_server,node_n_core_mcp,node_n_core_plugins toneBlue
class node_n_llm_client,node_n_llm_intents,node_n_llm_parser,node_n_llm_prompts toneAmber
class node_n_ui_pet,node_n_ui_tui toneMint
class node_n_mc_entry,node_n_mc_agent,node_n_mc_bridge toneRose
class node_n_plugin_obsidian,node_n_plugin_tui toneIndigo
```

## Components

### Core runtime

Shared state, sensing, orchestration, config, and local services.

- [Config](https://github.com/braedynl0530/desktoppetml/blob/master/core/config.py): settings/runtime · Environment-driven feature flags, modes, rates, and provider selection. · `core/config.py`
- [Tracking](https://github.com/braedynl0530/desktoppetml/blob/master/core/tracking.py): sensing · Desktop activity and user-state observation. · `core/tracking.py`
- [Memory](https://github.com/braedynl0530/desktoppetml/blob/master/core/memory.py): state/store · Durable behavior history and inferred state storage. · `core/memory.py`
- [Short memory](https://github.com/braedynl0530/desktoppetml/blob/master/core/short_memory.py): ephemeral state · Near-term context used for recent interactions. · `core/short_memory.py`
- [Tiered memory](https://github.com/braedynl0530/desktoppetml/blob/master/core/Tiered_memory.py): memory policy · Multi-level memory management and retention. · `core/Tiered_memory.py`
- [Personality](https://github.com/braedynl0530/desktoppetml/blob/master/core/personalityEngine.py): behavior engine · Mood, preference, and response shaping from tracked state. · `core/personalityEngine.py`
- [Preferences](https://github.com/braedynl0530/desktoppetml/blob/master/core/item_preferences.py): habit model · Item and preference inference used by the personality layer. · `core/item_preferences.py`
- [Speech input](https://github.com/braedynl0530/desktoppetml/blob/master/core/STT.py): stt · Speech-to-text capture and normalization. · `core/STT.py`
- [Messaging](https://github.com/braedynl0530/desktoppetml/blob/master/core/messaging.py): chat/output · User-facing message and bubble routing. · `core/messaging.py`
- [Agents](https://github.com/braedynl0530/desktoppetml/blob/master/core/agents.py): orchestration · Coordinates state, speech, and model output into actions. · `core/agents.py`
- [Agent bridge](https://github.com/braedynl0530/desktoppetml/blob/master/core/agent_bridge.py): command bridge · Transforms model decisions into executable actions. · `core/agent_bridge.py`
- [Local service](https://github.com/braedynl0530/desktoppetml/blob/master/core/local_agent_service.py): local api · Local HTTP/service interface for planning and summarization. · `core/local_agent_service.py`
- [Local server](https://github.com/braedynl0530/desktoppetml/blob/master/core/local_agent_server.py): local api · HTTP entry point for local agent tasks. · `core/local_agent_server.py`
- [MCP client](https://github.com/braedynl0530/desktoppetml/blob/master/core/mcp_client.py): tool bridge · External tool/service integration, especially note workflows. · `core/mcp_client.py`
- [Plugin system](https://github.com/braedynl0530/desktoppetml/blob/master/core/plugin_system.py): extensions · Loads and dispatches optional plugins. · `core/plugin_system.py`

### LLM layer

Model clients, intent mapping, and response shaping.

- [Ollama client](https://github.com/braedynl0530/desktoppetml/blob/master/llm/ollama_client.py): llm client · Local model access layer. · `llm/ollama_client.py`
- [Intents](https://github.com/braedynl0530/desktoppetml/blob/master/llm/intents.py): intent map · Maps language to commands or intent classes. · `llm/intents.py`
- [Response parser](https://github.com/braedynl0530/desktoppetml/blob/master/llm/response_parser.py): structured output · Converts model output into structured actions. · `llm/response_parser.py`
- [Prompts](https://github.com/braedynl0530/desktoppetml/tree/master/llm/prompts): prompt assets · System prompts for personality, reasoning, and Minecraft behavior. · `llm/prompts`

### Desktop UI

GUI and terminal presentation surfaces.

- [Desktop pet](https://github.com/braedynl0530/desktoppetml/blob/master/ui/pet.py): launcher/ui · Main desktop launcher and GUI entry point. · `ui/pet.py`
- [Terminal UI](https://github.com/braedynl0530/desktoppetml/blob/master/ui/tui.py): tui/ui · Standalone terminal mode for interaction. · `ui/tui.py`

### Minecraft bot

Scarpet entry point and bridge/agent logic for the fake-player side.

- [Petbot main](https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/petbot_main.sc): scarpet entry · Minecraft-side entry script for the fake player. · `minecraft/petbot_main.sc`
- [Minecraft agent](https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/minecraft_agent.py): game agent · In-game behavior, state handling, and action selection. · `minecraft/minecraft_agent.py`
- [Minecraft bridge](https://github.com/braedynl0530/desktoppetml/blob/master/minecraft/minecraft_bridge.py): http bridge · Communicates game context and actions between Minecraft and the core runtime. · `minecraft/minecraft_bridge.py`

### Plugins

Optional extensions loaded by the core.

- [Obsidian plugin](https://github.com/braedynl0530/desktoppetml/blob/master/core/plugins/obsidian_plugin.py): plugin · Note and MCP-oriented extension. · `core/plugins/obsidian_plugin.py`
- [TUI plugin](https://github.com/braedynl0530/desktoppetml/blob/master/core/plugins/tui_plugin.py): plugin · Terminal interaction extension. · `core/plugins/tui_plugin.py`

## Connections

- Desktop pet → Config: reads mode · Launcher selects GUI or TUI based on runtime settings.
- Desktop pet → Agents: drives · Desktop interaction is forwarded into the shared agent loop.
- Terminal UI → Agents: sends input · Terminal commands enter the same orchestration path.
- Tracking → Memory: persists · Observed desktop events are stored for later inference.
- Tracking → Short memory: updates · Recent context is kept in short-term memory.
- Short memory → Tiered memory: rolls up · Ephemeral context can be promoted into longer retention.
- Memory → Personality: feeds · Historical behavior informs mood and response decisions.
- Preferences → Personality: shapes · Preference inference adjusts personality output.
- Speech input → Agents: input · Speech input is normalized before orchestration.
- Agents → Intents: classifies · Orchestrator maps user language into intent categories.
- Agents → Ollama client: queries · Agent layer requests model inference.
- Ollama client → Prompts: uses · Model calls are guided by prompt assets.
- Ollama client → Response parser: returns text · Raw model output is parsed into structured actions.
- Response parser → Agent bridge: emits actions · Parsed model responses become executable commands.
- Agent bridge → Messaging: routes · Action results are surfaced to the user as messages.
- Agent bridge → Plugin system: dispatches · Bridge can hand commands or events to plugins.
- Plugin system → Obsidian plugin: loads · Optional note/MCP extension is registered.
- Plugin system → TUI plugin: loads · Optional terminal extension is registered.
- Local service → Local server: serves via · Service logic is exposed through a local HTTP server.
- Local service → Agents: supports · Planning and summarization outputs feed the agent pipeline.
- MCP client → Obsidian plugin: connects · MCP client underpins note-oriented workflows.
- Petbot main → Minecraft bridge: starts · Scarpet entry launches the Minecraft bridge stack.
- Minecraft bridge → Minecraft agent: coordinates · Game context is converted into in-game agent behavior.
- Minecraft bridge → Ollama client: calls · Minecraft decisions use the shared model layer.
- Minecraft bridge → Prompts: uses · Minecraft-specific prompting guides behavior.
- Minecraft agent → Response parser: interprets · Model output is converted into game actions.
- Minecraft agent → Memory: shares state · Minecraft-side history can reuse shared memory concepts.
- Config → Tracking: tunes · Polling, debounce, and environment behavior are configuration-driven.
- Config → Ollama client: selects provider · Model/provider choice is controlled centrally.
