How braedynl0530/desktoppetml works
1. Overall shape
This repo is a local, event-driven desktop companion with two execution surfaces: a desktop pet UI and a Minecraft fake-player bot. Both are coordinated by shared “core” services rather than by a single monolith.
2. Primary entry points
- ui/pet.py is the main launcher for the desktop experience and auto-selects GUI vs TUI.
- ui/tui.py is a standalone terminal mode.
- core/local_agent_server.py exposes a small local HTTP service for planning/summarization tasks.
- minecraft/petbot_main.sc is the Scarpet script entry point on the Minecraft side.
3. Core runtime and configuration
core/config.py centralizes feature flags, polling rates, model/provider selection, memory limits, and UI mode. The app is highly environment-variable driven, which means architecture and behavior change without code changes. core/platform_utils.py likely abstracts OS-specific behavior such as window detection and process checks.
4. Desktop sensing and behavior loop
The desktop pet monitors user activity through core/tracking.py, with memory and state handled by core/memory.py, core/short_memory.py, and core/Tiered_memory.py. core/item_preferences.py and core/personalityEngine.py turn those observations into mood, preference, and dialog decisions. This is the main feedback loop: observe apps/events → classify → store → infer behavior → react.
5. Speech and messaging
core/STT.py handles speech-to-text input, while core/messaging.py appears to route user-facing text/chat bubble messages. These inputs are normalized before reaching the LLM or command system, with debounce and retry controls coming from config.
6. LLM boundary
The llm/ package is the model-facing layer:
- llm/ollama_client.py is the local model client.
- llm/intents.py defines intent classification or command mapping.
- llm/response_parser.py turns model output into structured actions.
- llm/prompts/ contains system prompts for personality, reasoning, and Minecraft behavior.
Architecturally, this is the boundary between “natural language” and “actionable commands.”
7. Agent orchestration
core/agents.py and core/agent_bridge.py are the coordination layer that converts tracked state, STT, and LLM output into concrete actions. core/mcp_client.py and the MCP-related settings in config indicate an integration path for external tools/services, especially Obsidian. core/local_agent_service.py likely provides a small local service interface for note/planning outputs.
8. Plugin system
core/plugin_system.py loads optional plugins from core/plugins/. The visible plugins are:
- obsidian_plugin.py for Obsidian/MCP note workflows.
- tui_plugin.py for terminal interaction support.
Plugins are a clear extension boundary: they consume core events/commands without owning the main loop.
9. Minecraft subsystem
minecraft/minecraft_agent.py and minecraft/minecraft_bridge.py form a separate but connected behavior stack for the in-game fake player. The README indicates communication via an HTTP bridge, with the Minecraft side sending context such as chat, position, held items, and nearby blocks, and receiving actions back. This subsystem has its own personality, memory, and action vocabulary, but still depends on the shared LLM and core decision logic.
10. UI and presentation layer
ui/pet.py contains the GUI desktop pet, likely using PyQt5 for animations, chat bubbles, and mic controls. ui/tui.py provides the terminal UI and command entry path. The presentation layer is intentionally thin: it displays state and forwards user interaction into the core/LLM pipeline.
11. Data persistence and learning
The repository uses SQLite as the durable store for behavior history and model/tracking data. The learning path described in the README is: application/window events → persisted records → anomaly detection / habit modeling → personality and reaction adjustments. core/tracking.py and the memory modules are the main persistence-to-inference bridge.
12. External dependencies and system integration
Several outside systems materially affect the architecture:
- Ollama for local inference, with optional Gemini provider support.
- Fabric/Carpet and Scarpet for Minecraft automation.
- psutil for Minecraft process detection.
- xdotool or platform APIs for active-window detection.
- Optional Obsidian MCP transport for note operations.
- requirements.txt and installer stuff/hook-all.py support packaging/deployment.
13. Main data flows to diagram
- Desktop flow: user activity/STT → tracking/memory → LLM/intents → personality/messaging → UI animations/chat.
- Minecraft flow: game state/chat → bridge/agent → LLM/prompting → response parser → in-game action.
- Planning/notes flow: typed/STT command → plugin/MCP or local agent service → summarized/structured output.
These flows share the same core philosophy: event capture, stateful memory, LLM interpretation, then domain-specific action.
Main entry points: agents.py Agents · ollama_client.py Ollama client · minecraft_bridge.py Minecraft bridge · config.py Config
Components 26
Core runtime
Shared state, sensing, orchestration, config, and local services.
- Config · Environment-driven feature flags, modes, rates, and provider selection.
core/config.py - Tracking · Desktop activity and user-state observation.
core/tracking.py - Memory · Durable behavior history and inferred state storage.
core/memory.py - Short memory · Near-term context used for recent interactions.
core/short_memory.py - Tiered memory · Multi-level memory management and retention.
core/Tiered_memory.py - Personality · Mood, preference, and response shaping from tracked state.
core/personalityEngine.py - Preferences · Item and preference inference used by the personality layer.
core/item_preferences.py - Speech input · Speech-to-text capture and normalization.
core/STT.py - Messaging · User-facing message and bubble routing.
core/messaging.py - Agents · Coordinates state, speech, and model output into actions.
core/agents.py - Agent bridge · Transforms model decisions into executable actions.
core/agent_bridge.py - Local service · Local HTTP/service interface for planning and summarization.
core/local_agent_service.py - Local server · HTTP entry point for local agent tasks.
core/local_agent_server.py - MCP client · External tool/service integration, especially note workflows.
core/mcp_client.py - Plugin system · Loads and dispatches optional plugins.
core/plugin_system.py
LLM layer
Model clients, intent mapping, and response shaping.
- Ollama client · Local model access layer.
llm/ollama_client.py - Intents · Maps language to commands or intent classes.
llm/intents.py - Response parser · Converts model output into structured actions.
llm/response_parser.py - Prompts · System prompts for personality, reasoning, and Minecraft behavior.
llm/prompts
Desktop UI
GUI and terminal presentation surfaces.
- Desktop pet · Main desktop launcher and GUI entry point.
ui/pet.py - Terminal UI · Standalone terminal mode for interaction.
ui/tui.py
Minecraft bot
Scarpet entry point and bridge/agent logic for the fake-player side.
- Petbot main · Minecraft-side entry script for the fake player.
minecraft/petbot_main.sc - Minecraft agent · In-game behavior, state handling, and action selection.
minecraft/minecraft_agent.py - Minecraft bridge · Communicates game context and actions between Minecraft and the core runtime.
minecraft/minecraft_bridge.py
Plugins
Optional extensions loaded by the core.
- Obsidian plugin · Note and MCP-oriented extension.
core/plugins/obsidian_plugin.py - TUI plugin · Terminal interaction extension.
core/plugins/tui_plugin.py
Connections 29
- Desktop pet → Config: reads mode · Launcher selects GUI or TUI based on runtime settings.
- Desktop pet → Agents: drives · Desktop interaction is forwarded into the shared agent loop.
- Terminal UI → Agents: sends input · Terminal commands enter the same orchestration path.
- Tracking → Memory: persists · Observed desktop events are stored for later inference.
- Tracking → Short memory: updates · Recent context is kept in short-term memory.
- Short memory → Tiered memory: rolls up · Ephemeral context can be promoted into longer retention.
- Memory → Personality: feeds · Historical behavior informs mood and response decisions.
- Preferences → Personality: shapes · Preference inference adjusts personality output.
- Speech input → Agents: input · Speech input is normalized before orchestration.
- Agents → Intents: classifies · Orchestrator maps user language into intent categories.
- Agents → Ollama client: queries · Agent layer requests model inference.
- Ollama client → Prompts: uses · Model calls are guided by prompt assets.
- Ollama client → Response parser: returns text · Raw model output is parsed into structured actions.
- Response parser → Agent bridge: emits actions · Parsed model responses become executable commands.
- Agent bridge → Messaging: routes · Action results are surfaced to the user as messages.
- Agent bridge → Plugin system: dispatches · Bridge can hand commands or events to plugins.
- Plugin system → Obsidian plugin: loads · Optional note/MCP extension is registered.
- Plugin system → TUI plugin: loads · Optional terminal extension is registered.
- Local service → Local server: serves via · Service logic is exposed through a local HTTP server.
- Local service → Agents: supports · Planning and summarization outputs feed the agent pipeline.
- MCP client → Obsidian plugin: connects · MCP client underpins note-oriented workflows.
- Petbot main → Minecraft bridge: starts · Scarpet entry launches the Minecraft bridge stack.
- Minecraft bridge → Minecraft agent: coordinates · Game context is converted into in-game agent behavior.
- Minecraft bridge → Ollama client: calls · Minecraft decisions use the shared model layer.
- Minecraft bridge → Prompts: uses · Minecraft-specific prompting guides behavior.
- Minecraft agent → Response parser: interprets · Model output is converted into game actions.
- Minecraft agent → Memory: shares state · Minecraft-side history can reuse shared memory concepts.
- Config → Tracking: tunes · Polling, debounce, and environment behavior are configuration-driven.
- Config → Ollama client: selects provider · Model/provider choice is controlled centrally.