How ifuryst/open-codex-computer-use works
open-computer-use is a cross-platform computer-use service exposed through MCP, with CLI and JavaScript REPL entry points. Its core workflow is to discover applications, inspect accessibility state, dispatch a selected UI action, and return updated state. The README also documents permission onboarding, macOS cursor motion, and Linux/Windows runtimes. Some links reflect documented capabilities because their implementation excerpts were not sampled; optional peripheral wiring is omitted.
Main entry points: ComputerUseService.swift Computer Use Service · MCPAppRuntime.swift macOS Runtime · MCPServer.swift MCP Server · ComputerUseToolDispatcher.swift Tool Dispatcher
Components 20
Client Interfaces
- MCP Server
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift - Tool Dispatcher
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift - CLI
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift - JavaScript REPL
scripts/node-repl/open-computer-use-repl.mjs - REPL Kernel
scripts/node-repl/open-computer-use-kernel.mjs
Computer Use Core
- Computer Use Service
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift - App Discovery
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift - Accessibility Snapshot
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift - Input Simulation
packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift - Permission Onboarding
apps/OpenComputerUse/Sources/OpenComputerUse/PermissionOnboardingApp.swift
Platform Runtimes
- macOS Runtime
apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift - Linux Runtime
apps/OpenComputerUseLinux/main.go - Linux Accessibility
apps/OpenComputerUseLinux/runtime.py - Windows Runtime
apps/OpenComputerUseWindows/main.go
Cursor Motion
- Cursor Motion App
experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift - Cursor Motion Model
experiments/CursorMotion/Sources/CursorMotion/CursorMotionModel.swift - Standalone Cursor Viewer
experiments/StandaloneCursor/Sources/StandaloneCursor/StandaloneCursorRootView.swift - Cursor Motion Engine
experiments/StandaloneCursor/Sources/StandaloneCursorSupport/StandaloneCursorModel.swift
Other components
- AI Agent / MCP Client
- Desktop Accessibility / Input APIs
Connections 21
- AI Agent / MCP Client → MCP Server: sends MCP calls
- MCP Server → Tool Dispatcher: dispatches tools
- Tool Dispatcher → Computer Use Service: invokes operations
- CLI → Computer Use Service: invokes operations
- JavaScript REPL → REPL Kernel: evaluates code
- REPL Kernel → CLI: calls CLI
- Computer Use Service → App Discovery: lists apps
- Computer Use Service → Accessibility Snapshot: reads app state
- Computer Use Service → Input Simulation: simulates actions
- macOS Runtime → MCP Server: runs MCP service
- macOS Runtime → Permission Onboarding: checks permissions
- Computer Use Service → macOS Runtime: uses macOS runtime
- Computer Use Service → Linux Runtime: uses Linux runtime
- Linux Runtime → Linux Accessibility: runs accessibility requests
- Computer Use Service → Windows Runtime: uses Windows runtime
- Accessibility Snapshot → Desktop Accessibility / Input APIs: reads accessibility tree
- Input Simulation → Desktop Accessibility / Input APIs: sends input events
- Linux Accessibility → Desktop Accessibility / Input APIs: queries and controls desktop
- Windows Runtime → Desktop Accessibility / Input APIs: controls desktop
- Cursor Motion App → Cursor Motion Model: generates motion paths
- Standalone Cursor Viewer → Cursor Motion Engine: uses motion engine