How ghostwright/ghost-os works
Ghost OS is a macOS computer-use service for AI agents, exposed through MCP. The main workflow is an MCP client requesting perception or an action; tool dispatch routes to accessibility-based perception, optional visual grounding, and macOS input actions. Agents can also record manual workflows and replay parameterized JSON recipes. The graph preserves setup/diagnostics and screenshot support. Edges follow shown calls, source-index references, or README statements; unsampled implementations and sidecar details are not inferred beyond documented links.
Main entry points: MCPDispatch.swift Tool dispatch · RecipeEngine.swift Recipe execution · Perception.swift AX perception · Actions.swift Input actions
Components 22
Agent interface
- Ghost CLI
Sources/ghost/main.swift - MCP server
Sources/GhostOS/MCP/MCPServer.swift - Tool dispatch
Sources/GhostOS/MCP/MCPDispatch.swift
Perception
- AX perception
Sources/GhostOS/Perception/Perception.swift - Screen annotation
Sources/GhostOS/Perception/Annotate.swift - Screen capture
Sources/GhostOS/Screenshot/ScreenCapture.swift - CDP bridge
Sources/GhostOS/Vision/CDPBridge.swift - Visual grounding
Sources/GhostOS/Vision/VisionPerception.swift - Vision HTTP client
Sources/GhostOS/Vision/VisionBridge.swift - Vision sidecar
vision-sidecar/server.py
Computer control
- Input actions
Sources/GhostOS/Actions/Actions.swift - Focus manager
Sources/GhostOS/Actions/FocusManager.swift - Condition waits
Sources/GhostOS/MCP/WaitManager.swift
Learning and recipes
- Learning tools
Sources/GhostOS/Learning/LearningDispatch.swift - Action recorder
Sources/GhostOS/Learning/LearningRecorder.swift - Event handlers
Sources/GhostOS/Learning/EventHandlers.swift - App switch detector
Sources/GhostOS/Learning/AppSwitchDetector.swift - Recipe execution
Sources/GhostOS/Recipes/RecipeEngine.swift - Recipe storage
Sources/GhostOS/Recipes/RecipeStore.swift
Local support
- Setup wizard
Sources/ghost/SetupWizard.swift - Diagnostics
Sources/ghost/Doctor.swift
Other components
- MCP client
Connections 39
- MCP client → MCP server: sends MCP
README.md - Ghost CLI → MCP server: starts
Sources/ghost/main.swift - Ghost CLI → Setup wizard: runs
Sources/ghost/main.swift - Ghost CLI → Diagnostics: runs
Sources/ghost/main.swift - MCP server → Tool dispatch: routes tools
Sources/GhostOS/MCP/MCPServer.swift - Tool dispatch → AX perception: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Screen annotation: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Input actions: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Focus manager: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Condition waits: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Visual grounding: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Learning tools: calls
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Recipe execution: runs recipes
Sources/GhostOS/MCP/MCPDispatch.swift - Tool dispatch → Recipe storage: manages recipes
Sources/GhostOS/MCP/MCPDispatch.swift - AX perception → Screen capture: captures screen
Sources/GhostOS/Perception/Perception.swift - AX perception → CDP bridge: falls back
Sources/GhostOS/Perception/Perception.swift - AX perception → Visual grounding: falls back
Sources/GhostOS/Perception/Perception.swift - Screen annotation → AX perception: uses context
Sources/GhostOS/Perception/Annotate.swift - Screen annotation → Screen capture: captures screen
Sources/GhostOS/Perception/Annotate.swift - Visual grounding → Vision HTTP client: requests grounding
Sources/GhostOS/Vision/VisionPerception.swift - Visual grounding → Screen capture: captures screen
Sources/GhostOS/Vision/VisionPerception.swift - Vision HTTP client → Vision sidecar: HTTP inference
Sources/GhostOS/Vision/VisionBridge.swift - Input actions → Focus manager: focuses app
Sources/GhostOS/Actions/Actions.swift - Input actions → AX perception: locates elements
Sources/GhostOS/Actions/Actions.swift - Input actions → Visual grounding: uses grounding
Sources/GhostOS/Actions/Actions.swift - Learning tools → Action recorder: records actions
Sources/GhostOS/Learning/LearningDispatch.swift - Action recorder → Event handlers: handles events
Sources/GhostOS/Learning/LearningRecorder.swift - Action recorder → App switch detector: detects switches
Sources/GhostOS/Learning/LearningRecorder.swift - Recipe execution → Input actions: executes steps
Sources/GhostOS/Recipes/RecipeEngine.swift - Recipe execution → AX perception: checks context
Sources/GhostOS/Recipes/RecipeEngine.swift - Recipe execution → Focus manager: manages focus
Sources/GhostOS/Recipes/RecipeEngine.swift - Recipe execution → Condition waits: waits for condition
Sources/GhostOS/Recipes/RecipeEngine.swift - Recipe execution → Recipe storage: loads recipe
- MCP client → Learning tools: learns workflows
README.md - MCP client → Recipe execution: runs recipes
README.md - Setup wizard → Recipe storage: installs recipes
README.md - Diagnostics → Screen capture: checks permission
Sources/ghost/Doctor.swift - Diagnostics → Recipe storage: checks recipes
Sources/ghost/Doctor.swift - Diagnostics → Vision HTTP client: checks vision
Sources/ghost/Doctor.swift