# sandraschi/windows-computer-use-mcp architecture

> Architecture diagram of the GitHub repository sandraschi/windows-computer-use-mcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sandraschi/windows-computer-use-mcp
- Repository: https://github.com/sandraschi/windows-computer-use-mcp

## Overview

Windows Computer Use MCP provides agent-callable Windows UI automation, autonomous missions, and a web operator UI. The principal workflow is an agent or operator request entering through MCP or the dashboard, dispatching an automation tool, acting on the desktop, and optionally checking the result. Mission execution adds retries and verification; observation, safety, telemetry, and UI integration are distinct supporting capabilities. Several relationships below rely on the README or SOURCE INDEX; unsampled implementation details are not inferred.

Main entry points: [`src/windows_computer_use_mcp/tools/portmanteau_elements.py`](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_elements.py) (Element actions), [`src/windows_computer_use_mcp/task_engine.py`](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/task_engine.py) (Mission runner), [`web_sota/src/App.tsx`](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src/App.tsx) (Operator dashboard), [`src/windows_computer_use_mcp/main.py`](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/main.py) (MCP server entry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_server_main["MCP server entry<br/>[main.py]"]
  node_mcp_app["MCP app<br/>[app.py]"]
  node_transport["MCP transport<br/>[transport.py]"]
  node_api["HTTP API"]
  node_tool_registry["Tool registration<br/>[__init__.py]"]
end

subgraph group_automation["Automation workflows"]
  node_element_tool["Element actions"]
  node_mouse_tool["Mouse actions"]
  node_window_tool["Window actions"]
  node_task_engine["Mission runner<br/>[task_engine.py]"]
  node_assert_engine["Assertions<br/>[assert_engine.py]"]
  node_retry["Retry policy<br/>[retry.py]"]
  node_shortcut["App shortcuts<br/>[shortcut_engine.py]"]
  node_dialog["File dialogs<br/>[dialog_engine.py]"]
  node_safety["Safety controls<br/>[safety.py]"]
  node_tool_models["Tool models<br/>[models.py]"]
end

subgraph group_desktop["Desktop interaction"]
  node_dispatch["Input dispatch<br/>[dispatch.py]"]
  node_mouse_runtime["Win32 mouse<br/>[win32_mouse.py]"]
  node_keyboard_runtime["Win32 keyboard<br/>[win32_keyboard.py]"]
  node_window_runtime["Win32 windows<br/>[win32_window.py]"]
end

subgraph group_state["Observation and state"]
  node_desktop_capture["Desktop capture<br/>[capture.py]"]
  node_display_utils["Display utilities<br/>[display_utils.py]"]
  node_snapshot_store[("Snapshot state<br/>[snapshot_store.py]")]
  node_telemetry[("Action telemetry<br/>[telemetry.py]")]
end

subgraph group_operator["Operator interface"]
  node_web_frontend["Operator dashboard<br/>[App.tsx]"]
  node_web_api["Web backend<br/>[server.py]"]
  node_tauri_shell["Desktop shell<br/>[main.rs]"]
  node_mcp_setup["MCP onboarding<br/>[mcp_client.rs]"]
end

node_agent(("AI agent"))
node_operator_user(("Web operator"))
node_mcp_host(("MCP client"))

node_agent -->|"requests automation"| node_mcp_host
node_mcp_host -->|"starts server"| node_server_main
node_server_main -->|"runs transport"| node_transport
node_server_main -->|"loads tools"| node_tool_registry
node_tool_registry -.->|"registers tools"| node_mcp_app
node_mcp_app -.->|"serves HTTP"| node_api
node_mcp_host -->|"calls tools"| node_element_tool
node_agent -->|"submits mission"| node_task_engine
node_task_engine -->|"checks outcomes"| node_assert_engine
node_task_engine -->|"retries actions"| node_retry
node_task_engine -->|"uses snapshots"| node_snapshot_store
node_task_engine -->|"runs shortcuts"| node_shortcut
node_task_engine -->|"submits paths"| node_dialog
node_element_tool -->|"resolves dispatch"| node_dispatch
node_element_tool -->|"checks safety"| node_safety
node_element_tool -->|"reads snapshots"| node_snapshot_store
node_element_tool -->|"uses models"| node_tool_models
node_element_tool -->|"targets windows"| node_window_runtime
node_element_tool -->|"performs clicks"| node_mouse_runtime
node_mouse_tool -->|"sends input"| node_mouse_runtime
node_window_tool -->|"uses display data"| node_display_utils
node_shortcut -->|"sends keys"| node_keyboard_runtime
node_shortcut -->|"checks stability"| node_assert_engine
node_dispatch -.->|"dispatches input"| node_mouse_runtime
node_desktop_capture -->|"captures window"| node_window_runtime
node_assert_engine -->|"targets window"| node_window_runtime
node_operator_user -->|"uses dashboard"| node_web_frontend
node_web_frontend -.->|"requests backend"| node_web_api
node_web_frontend -.->|"runs as desktop app"| node_tauri_shell
node_tauri_shell -->|"registers clients"| node_mcp_setup
node_web_frontend -.->|"configures MCP"| node_mcp_host

click node_server_main "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/main.py"
click node_mcp_app "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/app.py"
click node_transport "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/transport.py"
click node_api "https://github.com/sandraschi/windows-computer-use-mcp/tree/master/src/windows_computer_use_mcp/api/v1/endpoints"
click node_tool_registry "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/__init__.py"
click node_element_tool "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_elements.py"
click node_mouse_tool "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_mouse.py"
click node_window_tool "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_windows.py"
click node_task_engine "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/task_engine.py"
click node_assert_engine "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/assert_engine.py"
click node_retry "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/retry.py"
click node_shortcut "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/shortcut_engine.py"
click node_dialog "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/dialog_engine.py"
click node_safety "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/safety.py"
click node_tool_models "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/models.py"
click node_dispatch "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/dispatch.py"
click node_mouse_runtime "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_mouse.py"
click node_keyboard_runtime "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_keyboard.py"
click node_window_runtime "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_window.py"
click node_desktop_capture "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/desktop_state/capture.py"
click node_display_utils "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/display_utils.py"
click node_snapshot_store "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/snapshot_store.py"
click node_telemetry "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/telemetry.py"
click node_web_frontend "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src/App.tsx"
click node_web_api "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/backend/server.py"
click node_tauri_shell "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src-tauri/src/main.rs"
click node_mcp_setup "https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src-tauri/src/mcp_client.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server_main,node_mcp_app,node_transport,node_api,node_tool_registry,node_mcp_host toneBlue
class node_element_tool,node_mouse_tool,node_window_tool,node_task_engine,node_assert_engine,node_retry,node_shortcut,node_dialog,node_safety,node_tool_models toneAmber
class node_dispatch,node_mouse_runtime,node_keyboard_runtime,node_window_runtime toneMint
class node_desktop_capture,node_display_utils,node_snapshot_store,node_telemetry toneRose
class node_web_frontend,node_web_api,node_tauri_shell,node_mcp_setup,node_agent,node_operator_user toneIndigo
```

## Components

### Entry points

- [MCP server entry](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/main.py): `src/windows_computer_use_mcp/main.py`
- [MCP app](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/app.py): `src/windows_computer_use_mcp/app.py`
- [MCP transport](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/transport.py): `src/windows_computer_use_mcp/transport.py`
- [HTTP API](https://github.com/sandraschi/windows-computer-use-mcp/tree/master/src/windows_computer_use_mcp/api/v1/endpoints): `src/windows_computer_use_mcp/api/v1/endpoints`
- [Tool registration](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/__init__.py): `src/windows_computer_use_mcp/tools/__init__.py`

### Automation workflows

- [Element actions](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_elements.py): `src/windows_computer_use_mcp/tools/portmanteau_elements.py`
- [Mouse actions](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_mouse.py): `src/windows_computer_use_mcp/tools/portmanteau_mouse.py`
- [Window actions](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/portmanteau_windows.py): `src/windows_computer_use_mcp/tools/portmanteau_windows.py`
- [Mission runner](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/task_engine.py): `src/windows_computer_use_mcp/task_engine.py`
- [Assertions](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/assert_engine.py): `src/windows_computer_use_mcp/assert_engine.py`
- [Retry policy](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/retry.py): `src/windows_computer_use_mcp/retry.py`
- [App shortcuts](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/shortcut_engine.py): `src/windows_computer_use_mcp/shortcut_engine.py`
- [File dialogs](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/dialog_engine.py): `src/windows_computer_use_mcp/dialog_engine.py`
- [Safety controls](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/safety.py): `src/windows_computer_use_mcp/safety.py`
- [Tool models](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/tools/models.py): `src/windows_computer_use_mcp/tools/models.py`

### Desktop interaction

- [Input dispatch](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/dispatch.py): `src/windows_computer_use_mcp/dispatch.py`
- [Win32 mouse](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_mouse.py): `src/windows_computer_use_mcp/win32_mouse.py`
- [Win32 keyboard](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_keyboard.py): `src/windows_computer_use_mcp/win32_keyboard.py`
- [Win32 windows](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/win32_window.py): `src/windows_computer_use_mcp/win32_window.py`

### Observation and state

- [Desktop capture](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/desktop_state/capture.py): `src/windows_computer_use_mcp/desktop_state/capture.py`
- [Display utilities](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/display_utils.py): `src/windows_computer_use_mcp/display_utils.py`
- [Snapshot state](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/snapshot_store.py): `src/windows_computer_use_mcp/snapshot_store.py`
- [Action telemetry](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/src/windows_computer_use_mcp/telemetry.py): `src/windows_computer_use_mcp/telemetry.py`

### Operator interface

- [Operator dashboard](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src/App.tsx): `web_sota/src/App.tsx`
- [Web backend](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/backend/server.py): `web_sota/backend/server.py`
- [Desktop shell](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src-tauri/src/main.rs): `web_sota/src-tauri/src/main.rs`
- [MCP onboarding](https://github.com/sandraschi/windows-computer-use-mcp/blob/master/web_sota/src-tauri/src/mcp_client.rs): `web_sota/src-tauri/src/mcp_client.rs`

### Other components

- **AI agent**
- **Web operator**
- **MCP client**

## Connections

- AI agent → MCP client: requests automation (evidence: `README.md`)
- MCP client → MCP server entry: starts server (evidence: `README.md`)
- MCP server entry → MCP transport: runs transport (evidence: `src/windows_computer_use_mcp/main.py`)
- MCP server entry → Tool registration: loads tools (evidence: `src/windows_computer_use_mcp/main.py`)
- Tool registration → MCP app: registers tools
- MCP app → HTTP API: serves HTTP (evidence: `README.md`)
- MCP client → Element actions: calls tools (evidence: `README.md`)
- AI agent → Mission runner: submits mission (evidence: `README.md`)
- Mission runner → Assertions: checks outcomes (evidence: `src/windows_computer_use_mcp/task_engine.py`)
- Mission runner → Retry policy: retries actions (evidence: `src/windows_computer_use_mcp/task_engine.py`)
- Mission runner → Snapshot state: uses snapshots (evidence: `src/windows_computer_use_mcp/task_engine.py`)
- Mission runner → App shortcuts: runs shortcuts (evidence: `src/windows_computer_use_mcp/task_engine.py`)
- Mission runner → File dialogs: submits paths (evidence: `src/windows_computer_use_mcp/task_engine.py`)
- Element actions → Input dispatch: resolves dispatch (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Element actions → Safety controls: checks safety (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Element actions → Snapshot state: reads snapshots (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Element actions → Tool models: uses models (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Element actions → Win32 windows: targets windows (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Element actions → Win32 mouse: performs clicks (evidence: `src/windows_computer_use_mcp/tools/portmanteau_elements.py`)
- Mouse actions → Win32 mouse: sends input (evidence: `src/windows_computer_use_mcp/tools/portmanteau_mouse.py`)
- Window actions → Display utilities: uses display data (evidence: `src/windows_computer_use_mcp/tools/portmanteau_windows.py`)
- App shortcuts → Win32 keyboard: sends keys (evidence: `src/windows_computer_use_mcp/shortcut_engine.py`)
- App shortcuts → Assertions: checks stability (evidence: `src/windows_computer_use_mcp/shortcut_engine.py`)
- Input dispatch → Win32 mouse: dispatches input
- Desktop capture → Win32 windows: captures window (evidence: `src/windows_computer_use_mcp/desktop_state/capture.py`)
- Assertions → Win32 windows: targets window (evidence: `src/windows_computer_use_mcp/assert_engine.py`)
- Web operator → Operator dashboard: uses dashboard (evidence: `README.md`)
- Operator dashboard → Web backend: requests backend
- Operator dashboard → Desktop shell: runs as desktop app (evidence: `README.md`)
- Desktop shell → MCP onboarding: registers clients (evidence: `web_sota/src-tauri/src/main.rs`)
- Operator dashboard → MCP client: configures MCP
