How asaripa3/computer-use-automation-system works
This repository is a computer-use automation system for legacy back-office applications. Its main workflow is: a model explores an application through a browser surface, records and compiles that work into a typed capability, and a deterministic replay engine executes the capability under policy, returning success, a business outcome, or a failure. Human handoff is an escalation path for steps requiring intervention. The README and source index establish the major layers and several dependencies; unsampled details are represented only where documented or referenced.
Main entry points: engine.py Replay engine · loop.py Discovery loop · tools.py Discovery tools · app.py ShareBase console
Components 27
Target application
- ShareBase console
src/sharebase/app.py - Member data
src/sharebase/data.py - Session handling
src/sharebase/session.py - Fault injection
src/sharebase/faults.py
Surface interface
- Surface protocol
src/surface/model.py - Browser surface
src/surface/browser.py - Page view
src/surface/view.py
Capability contract
- Capability schema
src/contract/capability.py - Locator ladder
src/contract/locator.py - Typed values
src/contract/values.py - Contract derivation
src/contract/derive.py - Run results
src/contract/result.py
Discovery and replay
- Discovery loop
src/explore/loop.py - Model adapter
src/explore/model.py - Discovery tools
src/explore/tools.py - Capability recorder
src/explore/recorder.py - Replay engine
src/replay/engine.py - Target resolution
src/replay/resolve.py - Replay evidence
src/replay/evidence.py
Safety and handoff
- Allowlist policy
src/policy/allowlist.py - Risk gating
src/policy/risk.py - Data redaction
src/policy/redaction.py - Session control
src/handoff/control.py - Intervention request
src/handoff/intervention.py - Operator channel
src/handoff/operator.py - Handoff coordinator
src/handoff/coordinator.py
Other components
- Caller or operator
Connections 35
- Caller or operator → Discovery loop: starts discovery
README.md - Discovery loop → Model adapter: queries model
src/explore/loop.py - Discovery loop → Discovery tools: dispatches tools
src/explore/loop.py - Discovery tools → Browser surface: acts through
src/explore/tools.py - Browser surface → Surface protocol: implements
src/surface/browser.py - Discovery tools → Allowlist policy: checks actions
src/explore/tools.py - Discovery tools → Capability recorder: records actions
src/explore/tools.py - Discovery loop → Capability recorder: drives recording
src/explore/loop.py - Capability recorder → Contract derivation: derives contract
src/explore/recorder.py - Contract derivation → Surface protocol: uses observations
src/contract/derive.py - Contract derivation → Locator ladder: builds locators
src/contract/derive.py - Capability schema → Locator ladder: contains locator schema
src/contract/capability.py - Capability schema → Typed values: uses typed values
src/contract/capability.py - Caller or operator → Replay engine: invokes capability
README.md - Replay engine → Capability schema: executes contract
src/replay/engine.py - Replay engine → Target resolution: resolves targets
src/replay/engine.py - Target resolution → Contract derivation: uses derivation
src/replay/resolve.py - Target resolution → Locator ladder: evaluates locators
src/replay/resolve.py - Replay engine → Surface protocol: observes and acts
src/replay/engine.py - Replay engine → Allowlist policy: checks destinations
src/replay/engine.py - Replay engine → Risk gating: gates risky steps
src/replay/engine.py - Replay engine → Data redaction: redacts run data
src/replay/engine.py - Replay engine → Session control: transfers control
src/replay/engine.py - Replay engine → Replay evidence: records evidence
src/replay/engine.py - Replay engine → Run results: returns result
src/replay/engine.py - Session control → Intervention request: raises request
src/handoff/control.py - Handoff coordinator → Session control: coordinates control
src/handoff/coordinator.py - Handoff coordinator → Operator channel: routes intervention
src/handoff/coordinator.py - Operator channel → Caller or operator: requests decision
README.md - ShareBase console → Member data: reads member records
src/sharebase/app.py - ShareBase console → Session handling: checks session
src/sharebase/app.py - ShareBase console → Fault injection: checks armed faults
src/sharebase/app.py - Browser surface → ShareBase console: drives application
README.md - Page view → Surface protocol: formats observations
src/surface/view.py - Replay evidence → Data redaction: redacts evidence
src/replay/evidence.py