How sm5190/interface-computer-use works
Interface Computer Use automates browser workflows by discovering a task with an LLM, compiling the observed trajectory into a typed capability, then replaying that capability deterministically under policy controls. Operators initiate discovery, compilation, or replay through the CLI. Replay and discovery share browser interaction, target grounding, checkpoint evaluation, safety gating, and evidence capture. The repository also includes LegacyBank, a synthetic banking UI target with member, account, and transfer operations. README capabilities include human approval and structured outcomes; the excerpts show policy approval gates and outcome/evidence handling, but do not expose the complete handoff wiring.
Main entry points: discovery/engine.py Discovery Engine · replay/engine.py Replay Engine · cli.py CLI · gate.py Action Gate
Components 20
Operator Entry
- CLI
src/cua/cli.py
Workflow Discovery
- Discovery Engine
src/cua/discovery/engine.py - Discovery Model
src/cua/discovery/gemini.py - Evidence Recorder
src/cua/evidence/recorder.py
Capability Artifacts
- Capability Compiler
src/cua/compiler/compiler.py - Typed Domain Models
src/cua/domain
Execution Runtime
- Replay Engine
src/cua/replay/engine.py - Session Bootstrap
src/cua/replay/bootstrap.py - Action Gate
src/cua/safety/gate.py - Application Policy
src/cua/safety/policy.py - Risk Classification
src/cua/safety/risk.py - Sensitive Data Redaction
src/cua/safety/redaction.py - Target Resolver
src/cua/grounding/resolver.py - Checkpoint Evaluator
src/cua/grounding/checkpoints.py - Playwright Surface
src/cua/surfaces/playwright_surface.py - Human Approval
src/cua/handoff
LegacyBank Target
- Banking Console
demo_app/app.py - Banking Database
demo_app/db.py
Other components
- Operator
- Browser
Connections 26
- Operator → CLI: runs commands
- CLI → Discovery Engine: starts discovery
- CLI → Capability Compiler: compiles trajectory
- CLI → Replay Engine: starts replay
- Discovery Engine → Discovery Model: requests decisions
- Discovery Engine → Playwright Surface: observes and acts
- Discovery Engine → Target Resolver: grounds targets
- Discovery Engine → Action Gate: checks actions
- Discovery Engine → Checkpoint Evaluator: evaluates effects
- Discovery Engine → Evidence Recorder: records run
- Capability Compiler → Typed Domain Models: builds artifact
- CLI → Typed Domain Models: validates inputs
- Replay Engine → Playwright Surface: observes and acts
- Replay Engine → Target Resolver: grounds targets
- Replay Engine → Action Gate: checks actions
- Replay Engine → Checkpoint Evaluator: evaluates effects
- Replay Engine → Session Bootstrap: initializes session
- Action Gate → Application Policy: consults rules
- Action Gate → Risk Classification: classifies risk
- Action Gate → Human Approval: requires approval
- Checkpoint Evaluator → Target Resolver: resolves controls
- Checkpoint Evaluator → Playwright Surface: reads page state
- Playwright Surface → Browser: controls browser
- Browser → Banking Console: loads and operates
- Banking Console → Banking Database: reads and writes
- Evidence Recorder → Sensitive Data Redaction: sanitizes records