How injaneity/pi-computer-use works
Pi Computer Use is a Pi extension that lets an agent discover desktop roots, observe and query interface state, then perform verified actions and wait for changes. The graph follows the documented tool workflow through the bridge to platform backends and native helpers, and includes browser-page support and state/outline handling. Platform helper internals are only partially sampled; edges to native implementations reflect the documented helper boundary and available excerpts, not an exhaustive account of each command.
Main entry points: bridge.ts Workflow bridge · index.ts Platform selection · computer-use.ts Tool registration · main.rs Windows command bridge
Components 23
Agent interface
- Tool registration
extensions/computer-use.ts
Workflow core
- Workflow bridge
src/bridge.ts - Configuration
src/config.ts - Saved states
src/state.ts - Resource scheduler
src/runtime.ts - Outline processing
src/outline.ts - Rendered changes
src/view.ts
Platform adapters
- Platform selection
src/platform/index.ts - macOS backend
src/platform/macos/backend.ts - Windows backend
src/platform/windows/backend.ts - Linux backend
src/platform/linux/backend.ts - macOS helper client
src/platform/macos/helper.ts - Windows helper client
src/platform/windows/helper.ts - Linux helper client
src/platform/linux/helper.ts
Native helpers
- macOS native bridge
native/macos/bridge.swift - Windows command bridge
native/windows/bridge-rs/src/main.rs - Windows UI access
native/windows/bridge-rs/src/uia.rs - Windows input
native/windows/bridge-rs/src/input.rs - Linux command bridge
native/linux/bridge-rs/src/main.rs - AT-SPI access
native/linux/bridge-rs/src/atspi.rs - X11 support
native/linux/bridge-rs/src/x11.rs
Browser integration
- Browser pages
src/cdp.ts
Other components
- Pi agent
Connections 22
- Pi agent → Tool registration: invokes tools
- Tool registration → Configuration: loads settings
- Tool registration → Workflow bridge: dispatches tools
- Workflow bridge → Platform selection: selects backend
- Workflow bridge → Saved states: stores states
- Workflow bridge → Resource scheduler: orders work
- Workflow bridge → Outline processing: processes outlines
- Workflow bridge → Rendered changes: renders changes
- Platform selection → macOS backend: routes macOS
- Platform selection → Windows backend: routes Windows
- Platform selection → Linux backend: routes Linux
- macOS backend → macOS helper client: sends commands
- Windows backend → Windows helper client: sends commands
- Linux backend → Linux helper client: sends commands
- macOS helper client → macOS native bridge: uses helper
- Windows helper client → Windows command bridge: uses helper
- Linux helper client → Linux command bridge: uses helper
- Windows command bridge → Windows UI access: queries UI
- Windows command bridge → Windows input: performs input
- Linux command bridge → AT-SPI access: queries UI
- Linux command bridge → X11 support: enriches roots
- Workflow bridge → Browser pages: queries pages