How 777genius/os-ai-computer-use works
OS AI Computer Use is a local desktop-automation agent with CLI and backend entry points, a provider-neutral agent core, OpenAI/Anthropic model adapters, and cross-platform OS drivers. The main workflow accepts a task, repeatedly requests model responses, executes requested computer tools, and returns assistant output with usage information. The README also documents Flutter chat, sessions, uploads, settings, and cost tracking. Backend WebSocket internals and several core modules were not excerpted, so uncertain implementation links are omitted or marked unverified.
Main entry points: app.py HTTP and WebSocket API · drivers.py Platform drivers · orchestrator.py Agent orchestrator · orchestrator_runner.py Legacy agent loop
Components 26
User entry points
- CLI task runner
packages/cli/src/os_ai_cli/main.py
Backend services
- HTTP and WebSocket API
packages/backend/src/os_ai_backend/app.py - WebSocket RPC
packages/backend/src/os_ai_backend - Chat sessions
packages/backend/src/os_ai_backend/sessions.py - File storage
packages/backend/src/os_ai_backend/files.py - Settings and metrics
packages/backend/src/os_ai_backend/settings.py
Agent runtime
- Agent orchestrator
packages/core/src/os_ai_core/orchestrator.py - Agent use case
packages/core/src/os_ai_core/application/use_cases/run_agent.py - Legacy agent loop
packages/core/src/os_ai_core/adapters/legacy/orchestrator_runner.py - Legacy LLM gateway
packages/core/src/os_ai_core/adapters/llm/legacy_gateway.py - Composite tool gateway
packages/core/src/os_ai_core/adapters/tools/composite_tool_gateway.py - Computer tool provider
packages/core/src/os_ai_core/adapters/tools/local_computer_provider.py - Tool registry
packages/core/src/os_ai_core/tools/registry.py - MCP tool integration
packages/mcp/src/os_ai_mcp/client/tool_provider.py
Model providers
- LLM client contract
packages/llm/src/os_ai_llm/interfaces.py - Anthropic adapter
packages/llm_anthropic/src/os_ai_llm_anthropic/adapters_anthropic.py - OpenAI adapter
packages/llm_openai/src/os_ai_llm_openai/adapters_openai.py
Desktop execution
- Computer actions
packages/core/src/os_ai_core/tools/computer.py - OS capability ports
packages/os/src/os_ai_os/ports - Platform drivers
packages/os/src/os_ai_os/platform/drivers.py - Linux driver
packages/os-linux/src/os_ai_os_linux/drivers.py - macOS driver
packages/os-macos/src/os_ai_os_macos/drivers.py - Windows driver
packages/os-windows/src/os_ai_os_windows/drivers.py
Other components
- User
- Local desktop
- Provider APIs
Connections 26
- User → CLI task runner: submits task
- CLI task runner → Agent orchestrator: runs agent
packages/cli/src/os_ai_cli/main.py - User → HTTP and WebSocket API: uses desktop UI
- HTTP and WebSocket API → WebSocket RPC: routes WebSocket
packages/backend/src/os_ai_backend/app.py - HTTP and WebSocket API → File storage: stores uploads
packages/backend/src/os_ai_backend/app.py - HTTP and WebSocket API → Settings and metrics: updates settings
packages/backend/src/os_ai_backend/app.py - WebSocket RPC → Chat sessions: dispatches sessions
- Agent orchestrator → Agent use case: executes use case
packages/core/src/os_ai_core/orchestrator.py - Agent orchestrator → Legacy agent loop: runs legacy path
packages/core/src/os_ai_core/orchestrator.py - Legacy agent loop → Legacy LLM gateway: requests responses
packages/core/src/os_ai_core/adapters/legacy/orchestrator_runner.py - Legacy agent loop → Composite tool gateway: executes calls
packages/core/src/os_ai_core/adapters/legacy/orchestrator_runner.py - Composite tool gateway → Computer tool provider: dispatches tools
- Computer tool provider → Tool registry: uses registry
- Legacy LLM gateway → LLM client contract: adapts client
packages/core/src/os_ai_core/adapters/llm/legacy_gateway.py - Anthropic adapter → LLM client contract: implements contract
packages/llm_anthropic/src/os_ai_llm_anthropic/adapters_anthropic.py - OpenAI adapter → LLM client contract: implements contract
packages/llm_openai/src/os_ai_llm_openai/adapters_openai.py - Anthropic adapter → Provider APIs: calls provider
- OpenAI adapter → Provider APIs: calls provider
- Composite tool gateway → MCP tool integration: may use tools
- Computer tool provider → Computer actions: provides actions
- Computer actions → Local desktop: controls desktop
- Computer actions → OS capability ports: uses OS capabilities
- Platform drivers → OS capability ports: implements ports
packages/os/src/os_ai_os/platform/drivers.py - Linux driver → Platform drivers: supplies driver
packages/os-linux/src/os_ai_os_linux/drivers.py - macOS driver → Platform drivers: supplies driver
- Windows driver → Platform drivers: supplies driver
packages/os-windows/src/os_ai_os_windows/drivers.py