How mithra2000/computer-use-automation works
This is a computer-use automation system for learning and replaying a read-only banking lookup. A user starts discovery or replay through the CLI: discovery observes the banking UI, obtains model proposals, authorizes and executes actions, verifies the member and savings data, then saves a typed capability; replay validates and executes that capability without model decisions. Policy enforcement, browser access, evidence capture, and optional human takeover support both paths. Some subsystem links reflect the README because their implementation was not excerpted.
Main entry points: engine.py Replay engine · cli.py Automation CLI · agent.py Discovery agent · browser.py Browser adapter
Components 17
Run orchestration
- Automation CLI
automation/cli.py - Local server
automation/local_server.py - Validated result
automation/models/contracts.py
Capability discovery
- Discovery agent
automation/discovery/agent.py - Model provider
automation/discovery/providers.py - Artifact builder
automation/artifacts/builder.py - Versioned capability
capabilities/get_savings_balance/v1.json
Controlled execution
- Replay engine
automation/replay/engine.py - Browser adapter
automation/surface/browser.py
Safety and evidence
- Typed contracts
automation/models/contracts.py - Policy engine
automation/policy/engine.py - Evidence recorder
automation/evidence/recorder.py - Human handoff
automation/handoff/manager.py
Banking application
- LedgerDesk app
mock_bank/app.py
Other components
- Operator
- LLM service
- Human operator
Connections 25
- Operator → Automation CLI: starts run
README.md - Automation CLI → Discovery agent: starts discovery
automation/cli.py - Automation CLI → Model provider: creates provider
automation/cli.py - Model provider → LLM service: requests proposals
README.md - Discovery agent → Model provider: requests decisions
automation/discovery/agent.py - Discovery agent → Browser adapter: observes and acts
automation/discovery/agent.py - Discovery agent → Artifact builder: builds capability
automation/discovery/agent.py - Artifact builder → Versioned capability: saves artifact
automation/artifacts/builder.py - Automation CLI → Replay engine: starts replay
automation/cli.py - Automation CLI → Local server: starts optional app
automation/cli.py - Local server → LedgerDesk app: launches app
automation/local_server.py - Replay engine → Versioned capability: loads artifact
README.md - Replay engine → Browser adapter: executes steps
automation/replay/engine.py - Replay engine → Policy engine: checks authorization
automation/replay/engine.py - Browser adapter → Policy engine: authorizes browser actions
automation/surface/browser.py - Browser adapter → LedgerDesk app: interacts through UI
README.md - Discovery agent → Evidence recorder: records run
automation/discovery/agent.py - Replay engine → Evidence recorder: records run
automation/replay/engine.py - Replay engine → Human handoff: requests intervention
automation/replay/engine.py - Human handoff → Human operator: requests takeover
automation/handoff/manager.py - Human handoff → Browser adapter: resumes session
automation/handoff/manager.py - Automation CLI → Validated result: returns run result
automation/cli.py - Discovery agent → Typed contracts: validates proposals
automation/discovery/agent.py - Artifact builder → Typed contracts: constructs capability
automation/artifacts/builder.py - Replay engine → Typed contracts: validates result
automation/replay/engine.py