How alibaba/page-agent works
This repo is a TypeScript monorepo for “Page Agent”: a client-side agent that controls web pages through DOM/text inspection and LLM-driven actions. The architecture is split into a small set of reusable libraries plus optional surfaces for browser extension, MCP, and docs/demo delivery.
1. Primary runtime path: in-page agent loop
- The main control loop lives in packages/core and packages/page-agent.
- A user instruction is sent to an LLM, the model returns tool use / action intent, and the agent executes those actions against the current page.
- This is the core “reason → act → observe → repeat” boundary that an architecture diagram should center on.
2. DOM and page interaction layer
- packages/page-controller is the low-level browser/page adapter.
- It handles DOM tree extraction, page info collection, action execution, visual masking/overlays, and framework-specific patches (react, antd).
- This package is the boundary between the agent logic and the live webpage.
3. Agent orchestration and prompt/tooling
- packages/core contains PageAgentCore, prompt templates, tool definitions, and utility logic like auto-fixing.
- It is the reasoning/orchestration layer that transforms model output into structured page actions via the page-controller layer.
- Tests in this package suggest it is a stable, reusable core rather than a UI surface.
4. LLM provider abstraction
- packages/llms wraps model client behavior, errors, constants, and shared utilities.
- It appears to provide provider-agnostic, OpenAI-compatible integration so the agent can “bring your own LLM.”
- This is an explicit external-service boundary in the system.
5. Public browser library / SDK
- packages/page-agent is the user-facing package that packages the agent for direct use in a webpage.
- The README’s one-line script and new PageAgent(...) usage map to this package.
- It exposes the simplest integration path: the agent runs directly in-page without a backend or headless browser.
6. Multi-page browser extension
- packages/extension is the optional Chrome extension surface for cross-tab / multi-page workflows.
- It is split into background, content, sidepanel, hub, and remote controller modules.
- This extension is the boundary that extends the single-page agent into browser-wide automation.
7. Extension internal communication
- The extension uses background/content scripts and dedicated controllers (TabsController, RemotePageController, MultiPageAgent).
- The sidepanel UI is the human control plane; the background scripts mediate tab state and page connections.
- The hub entrypoint and websocket bridge suggest an internal coordination channel for remote control and multi-tab orchestration.
8. Shared UI primitives
- packages/ui contains reusable panel components, motion CSS, i18n, and shared utilities.
- This package likely feeds the extension panel and any embedded control surfaces.
- It is a presentation layer only; it should be drawn separate from agent logic and page automation.
9. Website and documentation app
- packages/website is a separate Vite app for the public site, docs, and interactive demos.
- It includes marketing pages, multilingual docs, API reference content, code editor/console demos, and theme/language switching.
- This is not part of the runtime agent, but it is the main human-facing documentation and showcase surface.
10. MCP server / external agent bridge
- packages/mcp provides an MCP server/launcher and a bridge to the extension hub.
- This enables external agent clients to control the browser agent from outside the page/extension boundary.
- In a diagram, this should sit as an integration layer between external AI clients and the browser extension.
11. Data flow summary
- User input enters via in-page SDK, extension sidepanel, website demo, or an MCP client.
- The agent converts intent into model calls through packages/llms.
- The model’s output is translated by packages/core into tool calls.
- packages/page-controller executes those calls against the DOM and returns page state for the next reasoning step.
- In extension mode, background/content/hub modules mediate the page connection across tabs.
12. Build, release, and repository boundaries
- The repo is managed with package-level TypeScript/Vite builds, Vitest tests, and shared scripts under scripts/.
- GitHub Actions cover CI, demo deployment, and release automation.
- This supports a clear separation between reusable libraries, browser extension packaging, and the documentation website.
13. Important architectural distinction
- The project is designed for client-side web enhancement, not server-side browser automation.
- The core product lives in the page/extension runtime, with external services limited mainly to the chosen LLM provider and optional MCP integrations.
Main entry points: PageAgentCore.ts Core loop · PageController.ts Page controller · MultiPageAgent.ts Extension agent · main.tsx Website
Components 18
Agent runtime
In-page agent loop, page control, and model abstraction.
- Page Agent · User-facing in-page entrypoint that packages the agent for direct browser use.
packages/page-agent/src/PageAgent.ts - Core loop · Reason/act/observe control loop and tool dispatch for page automation.
packages/core/src/PageAgentCore.ts - LLM clients · Provider abstraction and OpenAI-compatible client layer.
packages/llms/src/OpenAIClient.ts - Page controller · Low-level DOM inspection and action execution boundary against live pages.
packages/page-controller/src/PageController.ts - DOM tree · Extracts page structure and metadata for the agent to reason over.
packages/page-controller/src/dom/index.ts - Actions · Structured page operations executed by the controller.
packages/page-controller/src/actions.ts - Masking · Visual masking and simulator overlays for sensitive or obstructive content.
packages/page-controller/src/mask/SimulatorMask.ts - Framework patches · React and Ant Design specific patches for DOM interaction robustness.
packages/page-controller/src/patches/react.ts
Browser extension
Multi-tab orchestration and human control surfaces.
- Extension agent · Browser-extension control layer for cross-tab and remote page workflows.
packages/extension/src/agent/MultiPageAgent.ts - BG + content · Background and content scripts that connect tabs, pages, and the extension runtime.
packages/extension/src/entrypoints/background.ts - Tabs controller · Tab state and coordination logic used by multi-page workflows.
packages/extension/src/agent/TabsController.ts - Remote controller · Remote page control path with background/content implementations.
packages/extension/src/agent/RemotePageController.ts - Hub · Websocket-backed hub entrypoint for remote orchestration and browser bridge traffic.
packages/extension/src/entrypoints/hub/App.tsx - Sidepanel · Human-facing extension panel for configuration, history, and control.
packages/extension/src/entrypoints/sidepanel/App.tsx - Shared UI · Reusable panel components, motion CSS, and localization shared by extension surfaces.
packages/ui/src/panel/Panel.ts
Docs and SDKs
Public SDK, MCP bridge, and website/demo delivery.
- MCP server · External-agent bridge that connects MCP clients to the browser agent hub.
packages/mcp/src/index.js - Website · Public docs, API reference, and interactive showcase site.
packages/website/src/main.tsx
Repo tooling
Shared build, release, and CI automation.
- Build scripts · Shared build, CI, versioning, and publish automation.
scripts/build.js
Connections 21
- Page Agent → Core loop: uses · Public SDK delegates to the central agent loop.
- Core loop → LLM clients: calls · Core sends prompts and receives model output through the model abstraction.
- Core loop → Page controller: drives · Core translates model intent into page actions via the controller boundary.
- Core loop → DOM tree: reads · Agent reasoning depends on DOM/page state snapshots.
- Core loop → Actions: emits · Structured tool output becomes executable page operations.
- Page controller → DOM tree: extracts · Controller produces page structure and metadata for reasoning.
- Page controller → Masking: applies · Controller manages masking and overlay behavior on the page.
- Page controller → Framework patches: patches · Framework-specific shims improve interaction reliability.
- Extension agent → Tabs controller: coordinates · Multi-page orchestration relies on tab management.
- Extension agent → Remote controller: routes · Remote page control is mediated through dedicated controllers.
- Extension agent → BG + content: connects · Extension agent communicates through background and content scripts.
- BG + content → Page controller: attaches · Content scripts bridge extension runtime to the live page controller.
- BG + content → Hub: bridges · Background/content path can forward traffic to the hub channel.
- Sidepanel → Shared UI: renders with · Extension control surface is built from shared UI primitives.
- Hub → MCP server: exposes to · MCP layer uses the hub as an external control bridge.
- Website → Core loop: documents · Docs and demos explain the agent core and its APIs.
- Website → Page Agent: shows · Site includes the simplest in-page integration path.
- Website → Extension agent: shows · Site documents browser extension workflows and features.
- Build scripts → Page Agent: builds · Repo automation packages and versions the SDK and packages.
- Build scripts → Website: deploys · Automation covers docs/demo build and deployment.
- Build scripts → Extension agent: packages · Automation supports extension build and release flows.