computeruseprotocol/typescript-sdk

How computeruseprotocol/typescript-sdk works

CUP is a TypeScript SDK and integration server for AI agents to inspect and operate desktop interfaces. The main workflow creates a session, selects a platform adapter, captures an accessibility tree, and returns a CUP envelope or compact text; callers can then search the captured tree or execute actions against its elements. The graph also includes the CLI and MCP entry points. Platform implementations are shown as distinct capture and action backends; unsampled implementation details and undocumented wiring are omitted.

Main entry points: executor.ts Action executor · index.ts Session API · router.ts Platform router · keys.ts Key parsing

Components 21

SDK and output

Session API src/index.ts
Envelope and serializers src/format.ts
Element search src/search.ts
CUP types src/types.ts

Tree capture

Platform router src/router.ts
Windows capture src/platforms/windows.ts
macOS capture src/platforms/macos.ts
Linux capture src/platforms/linux.ts
Web capture src/platforms/web.ts

Action execution

Action executor src/actions/executor.ts
Handler contract src/actions/handler.ts
Key parsing src/actions/keys.ts
Windows actions src/actions/windows.ts
macOS actions src/actions/macos.ts
Linux actions src/actions/linux.ts
Web actions src/actions/web.ts

Agent interfaces

CLI src/cli.ts
MCP stdio entry src/mcp/cli.ts
MCP server src/mcp/server.ts

Other components

SDK caller
Chromium via CDP

Connections 23

SDK caller → Session API: calls README.md
Session API → Platform router: selects adapter src/index.ts
Platform router → Windows capture: dispatches src/router.ts
Platform router → macOS capture: dispatches src/router.ts
Platform router → Linux capture: dispatches src/router.ts
Platform router → Web capture: dispatches src/router.ts
Session API → Envelope and serializers: builds output src/index.ts
Session API → Element search: searches tree src/index.ts
Session API → Action executor: executes actions src/index.ts
Action executor → Handler contract: uses contract src/actions/executor.ts
Action executor → Windows actions: dispatches src/actions/executor.ts
Action executor → macOS actions: dispatches src/actions/executor.ts
Action executor → Linux actions: dispatches src/actions/executor.ts
Action executor → Web actions: dispatches src/actions/executor.ts
Windows actions → Key parsing: uses parsing src/actions/windows.ts
macOS actions → Key parsing: uses parsing src/actions/macos.ts
Linux actions → Key parsing: uses parsing src/actions/linux.ts
Web actions → Key parsing: uses parsing src/actions/web.ts
CLI → Platform router: selects platform src/cli.ts
CLI → Envelope and serializers: formats output src/cli.ts
MCP stdio entry → MCP server: starts server src/mcp/cli.ts
MCP server → Session API: calls SDK src/mcp/server.ts
Web capture → Chromium via CDP: connects via CDP README.md

More TypeScript diagrams: openobserve/openobserve · qwikdev/qwik · markdown-it/markdown-it · desktop/desktop · snarktank/ralph

Drawn by GitDiagram on September 30, 2026 from the repository's file tree, README and source files. Read as Markdown ·