How energypantry/agent-browser-runtime works
1. What this repo is
- A compose-managed “real Chrome” runtime for AI agents.
- The architecture is built around a visible, lease-based browser workspace instead of ephemeral headless sessions.
2. Core system boundary: the Broker
- broker/ is the control plane: Node/Fastify over HTTP + WebSocket.
- It owns leases, jobs, artifacts, pacing/cooldowns, session state, and health reporting.
- The broker is the main API surface other components talk to.
3. Browser execution boundary: Chrome runtime
- runtime/chrome/ packages Chromium/Chrome in Docker with Xvfb, x11vnc, noVNC, and startup scripts.
- This is the actual browser execution environment, with a persistent profile mount so logins, cookies, localStorage, and extension state survive restarts.
- Default host CDP port is 19223, which is part of the browser-control boundary.
4. Companion browser extension
- extension/ is the in-browser agent bridge.
- It owns real tabs and Tab Groups, can run debugger/CDP-style actions, capture screenshots/HTML, probe session state, and expose UI primitives such as move/click/type/scroll/wait.
- It is the mechanism that lets the system operate through visible browser UI rather than synthetic backend shortcuts.
5. Browser-to-control-plane flow
- The browser runtime loads the extension and exposes the live tab/session context.
- The broker coordinates actions and state, while the extension performs the actual browser-side operations.
- This split is important: the broker orchestrates; the extension executes inside Chrome.
6. Optional TLS gateway path
- tls-gateway/ is a separate Go service used as a local upstream proxy for Chromium’s startup networking path.
- It is opt-in by default and surfaced in broker status when enabled.
- Its role is transport consistency and proxy-chain control, not general application logic.
7. Client/operator interface
- cli/brs.js is the operator client for the whole system.
- It talks to the broker for status, one-shot fetches, continuous browsing leases, session probes, extractor runs, job inspection, artifact download, and cleanup.
- This CLI is the easiest way to see the architecture from the outside: it exercises the broker API directly.
8. Workload model: fetch, browse leases, and extractors
- The system supports two main browsing modes: one-shot fetch and persistent browse-* leases for multi-step workflows.
- extractors/ contains packaged site workflows that run against rendered pages and the broker/UI primitive layer.
- Extractors return structured results while also allowing artifacts like HTML and screenshots to be captured for evidence and debugging.
9. Artifacts and runtime state
- artifacts/, data/, and runtime/profile/ are the important persistence boundaries.
- Artifacts store outputs such as HTML, screenshots, probe results, and error evidence.
- Runtime profile state preserves the browser identity/session between runs; this is central to the repo’s design.
10. Deployment topology
- docker-compose.yml assembles the broker, browser runtime, and TLS gateway into one local stack.
- docker-compose.fingerprint.yml is an overlay for swapping in a mounted fingerprinted Chromium binary.
- scripts/smoke-test.sh validates the integrated stack end-to-end.
11. Operational behavior and human handoff
- noVNC is a first-class human handoff channel for login, Captcha, MFA, and other account-safety checkpoints.
- The repo emphasizes “real browser” pacing and UI interaction discipline rather than API shortcuts.
- Platform cooldowns and humanized pacing are part of the runtime behavior, not just application logic.
12. Documentation and usage contracts
- docs/SPEC.md, docs/broker-api.md, and docs/docker-compose-architecture.md define the intended contracts and deployment shape.
- skills/ contains Codex and OpenClaw skill packages describing how agents should use or deploy the runtime.
- These docs are part of the architecture because they encode the intended control flow and boundaries between broker, browser, and operator.
Main entry points: server.js Broker · entrypoint.sh Chrome runtime · background.js Extension · docker-compose.yml Compose stack
Components 21
Control plane
Broker, operator client, and workflow contracts
- CLI · Operator entrypoint for broker status, fetch, browse, probes, extractors, and cleanup
cli/brs.js - Broker · Fastify HTTP/WebSocket service that owns leases, jobs, artifacts, pacing, sessions, and health
broker/src/server.js - State store · Broker persistence for leases, jobs, runtime identity, and session data
broker/src/store.js - Extension RPC · Broker-side adapter for browser extension messaging and UI primitive execution
broker/src/extension-rpc.js - Healthcheck · Broker health and readiness reporting
broker/src/healthcheck.js - Extractors · Packaged site workflows that run against rendered pages and capture structured results
extractors/example.extract.js
Browser runtime
Chrome container, extension bridge, and browser-side execution
- Chrome runtime · Dockerized Chromium environment with persistent profile and visible browser session
runtime/chrome/entrypoint.sh - Launch script · Browser startup orchestration and runtime wiring
runtime/chrome/launch-browser.sh - Supervisor · Keeps browser-side services running inside the runtime container
runtime/chrome/supervisord.conf - Extension · In-browser agent layer for tabs, tab groups, screenshots, HTML, and UI primitives
extension/background.js - Runtime config · Extension-side runtime configuration and environment wiring
extension/runtime-config.js - Stealth script · Content script for browser-side stealth and page adaptation
extension/stealth-content.js - TLS gateway · Optional Go proxy for Chromium startup networking and proxy-chain control
tls-gateway/tls-gateway.go
Operations
Persistence, deployment, and validation
- Artifacts · Captured HTML, screenshots, probes, and error evidence
artifacts/.gitkeep - Data · Shared runtime data and persisted operational state
data/.gitkeep - Browser profile · Mounted browser identity, cookies, localStorage, and extension state
- Compose stack · Local stack assembly for broker, browser runtime, and optional TLS gateway
docker-compose.yml - Fingerprint overlay · Alternate compose overlay for fingerprinted Chromium binary
docker-compose.fingerprint.yml - Smoke test · End-to-end stack validation script
scripts/smoke-test.sh
Connections 28
- CLI → Broker: API · Operator client exercises the broker control surface
- Broker → State store: persists · Broker owns leases, jobs, sessions, and pacing state
- Broker → Extension RPC: commands · Broker delegates browser-side actions through extension RPC
- Broker → Healthcheck: reports · Broker exposes readiness and health status
- Broker → Artifacts: writes · Broker stores captured outputs and evidence
- Broker → Data: reads/writes · Broker uses shared operational state and runtime data
- Broker → Extractors: runs · Broker coordinates one-shot and lease-based workflows
- Broker → TLS gateway: uses · Broker surfaces optional gateway status when enabled
- Chrome runtime → Launch script: starts · Container entrypoint launches the browser stack
- Chrome runtime → Supervisor: supervises · Runtime container keeps browser-side services alive
- Chrome runtime → Extension: loads · Chrome runtime loads the agent bridge extension
- Chrome runtime → Browser profile: mounts · Persistent browser profile survives restarts
- Chrome runtime → TLS gateway: routes · Optional startup networking can flow through the gateway
- Extension → Runtime config: config · Extension behavior is parameterized by runtime config
- Extension → Stealth script: injects · Content script adapts page behavior inside browser tabs
- Extension → Artifacts: captures · Extension produces screenshots, HTML, and probe evidence
- Extension → State store: syncs · Extension state is coordinated with broker session state
- Extractors → Extension: uses · Packaged workflows rely on browser-side UI primitives
- Extractors → Broker: invoked by · Extractors run under broker-managed workflows
- Compose stack → Broker: assembles · Compose brings up the control plane
- Compose stack → Chrome runtime: assembles · Compose brings up the browser runtime
- Compose stack → TLS gateway: optionally includes · TLS gateway is part of the local stack when enabled
- Fingerprint overlay → Compose stack: overlays · Fingerprint stack modifies the base compose topology
- Smoke test → Compose stack: validates · Smoke test checks the integrated stack end to end
- Docs → Broker: specifies · Docs define broker API and contracts
- Docs → Compose stack: describes · Docs explain the deployment shape
- Skills → Broker: guides · Skills describe how operators use the runtime via broker
- Skills → Chrome runtime: guides · Skills describe deployment and browser runtime usage