How sandraschi/windows-computer-use-mcp works
Windows Computer Use MCP provides agent-callable Windows UI automation, autonomous missions, and a web operator UI. The principal workflow is an agent or operator request entering through MCP or the dashboard, dispatching an automation tool, acting on the desktop, and optionally checking the result. Mission execution adds retries and verification; observation, safety, telemetry, and UI integration are distinct supporting capabilities. Several relationships below rely on the README or SOURCE INDEX; unsampled implementation details are not inferred.
Main entry points: portmanteau_elements.py Element actions · task_engine.py Mission runner · App.tsx Operator dashboard · main.py MCP server entry
Components 30
Entry points
- MCP server entry
src/windows_computer_use_mcp/main.py - MCP app
src/windows_computer_use_mcp/app.py - MCP transport
src/windows_computer_use_mcp/transport.py - HTTP API
src/windows_computer_use_mcp/api/v1/endpoints - Tool registration
src/windows_computer_use_mcp/tools/__init__.py
Automation workflows
- Element actions
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Mouse actions
src/windows_computer_use_mcp/tools/portmanteau_mouse.py - Window actions
src/windows_computer_use_mcp/tools/portmanteau_windows.py - Mission runner
src/windows_computer_use_mcp/task_engine.py - Assertions
src/windows_computer_use_mcp/assert_engine.py - Retry policy
src/windows_computer_use_mcp/retry.py - App shortcuts
src/windows_computer_use_mcp/shortcut_engine.py - File dialogs
src/windows_computer_use_mcp/dialog_engine.py - Safety controls
src/windows_computer_use_mcp/safety.py - Tool models
src/windows_computer_use_mcp/tools/models.py
Desktop interaction
- Input dispatch
src/windows_computer_use_mcp/dispatch.py - Win32 mouse
src/windows_computer_use_mcp/win32_mouse.py - Win32 keyboard
src/windows_computer_use_mcp/win32_keyboard.py - Win32 windows
src/windows_computer_use_mcp/win32_window.py
Observation and state
- Desktop capture
src/windows_computer_use_mcp/desktop_state/capture.py - Display utilities
src/windows_computer_use_mcp/display_utils.py - Snapshot state
src/windows_computer_use_mcp/snapshot_store.py - Action telemetry
src/windows_computer_use_mcp/telemetry.py
Operator interface
- Operator dashboard
web_sota/src/App.tsx - Web backend
web_sota/backend/server.py - Desktop shell
web_sota/src-tauri/src/main.rs - MCP onboarding
web_sota/src-tauri/src/mcp_client.rs
Other components
- AI agent
- Web operator
- MCP client
Connections 31
- AI agent → MCP client: requests automation
README.md - MCP client → MCP server entry: starts server
README.md - MCP server entry → MCP transport: runs transport
src/windows_computer_use_mcp/main.py - MCP server entry → Tool registration: loads tools
src/windows_computer_use_mcp/main.py - Tool registration → MCP app: registers tools
- MCP app → HTTP API: serves HTTP
README.md - MCP client → Element actions: calls tools
README.md - AI agent → Mission runner: submits mission
README.md - Mission runner → Assertions: checks outcomes
src/windows_computer_use_mcp/task_engine.py - Mission runner → Retry policy: retries actions
src/windows_computer_use_mcp/task_engine.py - Mission runner → Snapshot state: uses snapshots
src/windows_computer_use_mcp/task_engine.py - Mission runner → App shortcuts: runs shortcuts
src/windows_computer_use_mcp/task_engine.py - Mission runner → File dialogs: submits paths
src/windows_computer_use_mcp/task_engine.py - Element actions → Input dispatch: resolves dispatch
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Element actions → Safety controls: checks safety
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Element actions → Snapshot state: reads snapshots
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Element actions → Tool models: uses models
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Element actions → Win32 windows: targets windows
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Element actions → Win32 mouse: performs clicks
src/windows_computer_use_mcp/tools/portmanteau_elements.py - Mouse actions → Win32 mouse: sends input
src/windows_computer_use_mcp/tools/portmanteau_mouse.py - Window actions → Display utilities: uses display data
src/windows_computer_use_mcp/tools/portmanteau_windows.py - App shortcuts → Win32 keyboard: sends keys
src/windows_computer_use_mcp/shortcut_engine.py - App shortcuts → Assertions: checks stability
src/windows_computer_use_mcp/shortcut_engine.py - Input dispatch → Win32 mouse: dispatches input
- Desktop capture → Win32 windows: captures window
src/windows_computer_use_mcp/desktop_state/capture.py - Assertions → Win32 windows: targets window
src/windows_computer_use_mcp/assert_engine.py - Web operator → Operator dashboard: uses dashboard
README.md - Operator dashboard → Web backend: requests backend
- Operator dashboard → Desktop shell: runs as desktop app
README.md - Desktop shell → MCP onboarding: registers clients
web_sota/src-tauri/src/main.rs - Operator dashboard → MCP client: configures MCP