# ifuryst/open-codex-computer-use architecture

> Architecture diagram of the GitHub repository ifuryst/open-codex-computer-use, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ifuryst/open-codex-computer-use
- Repository: https://github.com/ifuryst/open-codex-computer-use

## Overview

open-computer-use is a cross-platform computer-use service exposed through MCP, with CLI and JavaScript REPL entry points. Its core workflow is to discover applications, inspect accessibility state, dispatch a selected UI action, and return updated state. The README also documents permission onboarding, macOS cursor motion, and Linux/Windows runtimes. Some links reflect documented capabilities because their implementation excerpts were not sampled; optional peripheral wiring is omitted.

Main entry points: [`packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift) (Computer Use Service), [`apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift`](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift) (macOS Runtime), [`packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift) (MCP Server), [`packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift) (Tool Dispatcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Client Interfaces"]
  node_mcp["MCP Server<br/>[MCPServer.swift]"]
  node_dispatcher["Tool Dispatcher"]
  node_cli["CLI"]
  node_jsrepl["JavaScript REPL"]
  node_kernel["REPL Kernel"]
end

subgraph group_core["Computer Use Core"]
  node_service["Computer Use Service"]
  node_discovery["App Discovery<br/>[AppDiscovery.swift]"]
  node_snapshot["Accessibility Snapshot"]
  node_input["Input Simulation"]
  node_permissions["Permission Onboarding"]
end

subgraph group_platforms["Platform Runtimes"]
  node_macapp["macOS Runtime"]
  node_linux["Linux Runtime<br/>[main.go]"]
  node_linuxpy["Linux Accessibility<br/>[runtime.py]"]
  node_windows["Windows Runtime<br/>[main.go]"]
end

subgraph group_cursor["Cursor Motion"]
  node_cursorapp["Cursor Motion App"]
  node_cursormodel["Cursor Motion Model"]
  node_standalone["Standalone Cursor Viewer"]
  node_standalonemodel["Cursor Motion Engine"]
end

node_agent(("AI Agent / MCP Client"))
node_osapi["Desktop Accessibility / Input APIs"]

node_agent -->|"sends MCP calls"| node_mcp
node_mcp -->|"dispatches tools"| node_dispatcher
node_dispatcher -->|"invokes operations"| node_service
node_cli -->|"invokes operations"| node_service
node_jsrepl -->|"evaluates code"| node_kernel
node_kernel -->|"calls CLI"| node_cli
node_service -->|"lists apps"| node_discovery
node_service -->|"reads app state"| node_snapshot
node_service -->|"simulates actions"| node_input
node_macapp -->|"runs MCP service"| node_mcp
node_macapp -->|"checks permissions"| node_permissions
node_service -->|"uses macOS runtime"| node_macapp
node_service -.->|"uses Linux runtime"| node_linux
node_linux -->|"runs accessibility requests"| node_linuxpy
node_service -.->|"uses Windows runtime"| node_windows
node_snapshot -->|"reads accessibility tree"| node_osapi
node_input -->|"sends input events"| node_osapi
node_linuxpy -.->|"queries and controls desktop"| node_osapi
node_windows -.->|"controls desktop"| node_osapi
node_cursorapp -->|"generates motion paths"| node_cursormodel
node_standalone -->|"uses motion engine"| node_standalonemodel

click node_mcp "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift"
click node_dispatcher "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift"
click node_cli "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift"
click node_jsrepl "https://github.com/ifuryst/open-codex-computer-use/blob/main/scripts/node-repl/open-computer-use-repl.mjs"
click node_kernel "https://github.com/ifuryst/open-codex-computer-use/blob/main/scripts/node-repl/open-computer-use-kernel.mjs"
click node_service "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift"
click node_discovery "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift"
click node_snapshot "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift"
click node_input "https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift"
click node_permissions "https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUse/Sources/OpenComputerUse/PermissionOnboardingApp.swift"
click node_macapp "https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift"
click node_linux "https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseLinux/main.go"
click node_linuxpy "https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseLinux/runtime.py"
click node_windows "https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseWindows/main.go"
click node_cursorapp "https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift"
click node_cursormodel "https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/CursorMotion/Sources/CursorMotion/CursorMotionModel.swift"
click node_standalone "https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/StandaloneCursor/Sources/StandaloneCursor/StandaloneCursorRootView.swift"
click node_standalonemodel "https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/StandaloneCursor/Sources/StandaloneCursorSupport/StandaloneCursorModel.swift"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mcp,node_dispatcher,node_cli,node_jsrepl,node_kernel,node_agent toneBlue
class node_service,node_discovery,node_snapshot,node_input,node_permissions toneAmber
class node_macapp,node_linux,node_linuxpy,node_windows,node_osapi toneMint
class node_cursorapp,node_cursormodel,node_standalone,node_standalonemodel toneRose
```

## Components

### Client Interfaces

- [MCP Server](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/MCPServer.swift`
- [Tool Dispatcher](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseToolDispatcher.swift`
- [CLI](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/OpenComputerUseCLI.swift`
- [JavaScript REPL](https://github.com/ifuryst/open-codex-computer-use/blob/main/scripts/node-repl/open-computer-use-repl.mjs): `scripts/node-repl/open-computer-use-repl.mjs`
- [REPL Kernel](https://github.com/ifuryst/open-codex-computer-use/blob/main/scripts/node-repl/open-computer-use-kernel.mjs): `scripts/node-repl/open-computer-use-kernel.mjs`

### Computer Use Core

- [Computer Use Service](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/ComputerUseService.swift`
- [App Discovery](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AppDiscovery.swift`
- [Accessibility Snapshot](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/AccessibilitySnapshot.swift`
- [Input Simulation](https://github.com/ifuryst/open-codex-computer-use/blob/main/packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift): `packages/OpenComputerUseKit/Sources/OpenComputerUseKit/InputSimulation.swift`
- [Permission Onboarding](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUse/Sources/OpenComputerUse/PermissionOnboardingApp.swift): `apps/OpenComputerUse/Sources/OpenComputerUse/PermissionOnboardingApp.swift`

### Platform Runtimes

- [macOS Runtime](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift): `apps/OpenComputerUse/Sources/OpenComputerUse/MCPAppRuntime.swift`
- [Linux Runtime](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseLinux/main.go): `apps/OpenComputerUseLinux/main.go`
- [Linux Accessibility](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseLinux/runtime.py): `apps/OpenComputerUseLinux/runtime.py`
- [Windows Runtime](https://github.com/ifuryst/open-codex-computer-use/blob/main/apps/OpenComputerUseWindows/main.go): `apps/OpenComputerUseWindows/main.go`

### Cursor Motion

- [Cursor Motion App](https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift): `experiments/CursorMotion/Sources/CursorMotion/CursorMotionApp.swift`
- [Cursor Motion Model](https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/CursorMotion/Sources/CursorMotion/CursorMotionModel.swift): `experiments/CursorMotion/Sources/CursorMotion/CursorMotionModel.swift`
- [Standalone Cursor Viewer](https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/StandaloneCursor/Sources/StandaloneCursor/StandaloneCursorRootView.swift): `experiments/StandaloneCursor/Sources/StandaloneCursor/StandaloneCursorRootView.swift`
- [Cursor Motion Engine](https://github.com/ifuryst/open-codex-computer-use/blob/main/experiments/StandaloneCursor/Sources/StandaloneCursorSupport/StandaloneCursorModel.swift): `experiments/StandaloneCursor/Sources/StandaloneCursorSupport/StandaloneCursorModel.swift`

### Other components

- **AI Agent / MCP Client**
- **Desktop Accessibility / Input APIs**

## Connections

- AI Agent / MCP Client → MCP Server: sends MCP calls
- MCP Server → Tool Dispatcher: dispatches tools
- Tool Dispatcher → Computer Use Service: invokes operations
- CLI → Computer Use Service: invokes operations
- JavaScript REPL → REPL Kernel: evaluates code
- REPL Kernel → CLI: calls CLI
- Computer Use Service → App Discovery: lists apps
- Computer Use Service → Accessibility Snapshot: reads app state
- Computer Use Service → Input Simulation: simulates actions
- macOS Runtime → MCP Server: runs MCP service
- macOS Runtime → Permission Onboarding: checks permissions
- Computer Use Service → macOS Runtime: uses macOS runtime
- Computer Use Service → Linux Runtime: uses Linux runtime
- Linux Runtime → Linux Accessibility: runs accessibility requests
- Computer Use Service → Windows Runtime: uses Windows runtime
- Accessibility Snapshot → Desktop Accessibility / Input APIs: reads accessibility tree
- Input Simulation → Desktop Accessibility / Input APIs: sends input events
- Linux Accessibility → Desktop Accessibility / Input APIs: queries and controls desktop
- Windows Runtime → Desktop Accessibility / Input APIs: controls desktop
- Cursor Motion App → Cursor Motion Model: generates motion paths
- Standalone Cursor Viewer → Cursor Motion Engine: uses motion engine