How mend3/browser-use works
Project type and purpose
• This is an open-source Python library that lets AI agents control real browsers. It exposes an Agent API that wraps an LLM (e.g. OpenAI, Claude) and drives Playwright under the hood, with DOM extraction, action controllers, telemetry, and optional UI integrations.
a. External LLM Service
• OpenAI, Claude, Gemini, Bedrock, etc.
• Communicates over HTTP/HTTPS (API calls).
b. Agent Layer (browser_use/agent)
• Agent core: orchestrates task, system + user prompts, planning loops.
• Message Manager: splits LLM replies into discrete action requests.
• Prompt templates & system_prompt.md.
c. Controller Layer (browser_use/controller)
• Registry of actions (click, fill, screenshot, custom functions).
• Service + views: decouple HTTP or UI bindings from core logic.
d. Browser Executor (browser_use/browser)
• Playwright wrapper (Browser, Context).
• Executes low-level commands, takes screenshots/GIFs.
e. DOM Processor (browser_use/dom)
• buildDomTree.js + Python service: extracts DOM state for the LLM.
• History tree processor: tracks mutations across steps.
f. Telemetry & Logging
• browser_use/telemetry: collects metrics, events.
• logging_config.py: central logging.
g. Integration & UI Examples (examples/ui, examples/integrations)
• Gradio, Streamlit front-ends.
• Slack/Discord bots.
h. Persistence & Artifacts
• GIF outputs, logs, custom-function outputs (files).
Interactions and data flow
1. The user creates an Agent(task, llm=ChatOpenAI…)
2. Agent composes a system+user prompt and calls the LLM API
3. LLM returns textual plan with actions
4. Message Manager parses the plan into discrete action messages
5. Controller looks up each action in its registry and invokes corresponding service
6. Browser Executor runs Playwright commands, updates the DOM state
7. DOM Processor extracts the updated DOM and converts it into a structured representation
8. Agent sends updated context back to LLM for next iteration
9. Telemetry logs each step; screenshots/GIFs optionally generated
10. Final result returned to user
Architectural patterns and principles
• Layered separation: Agent → Message Manager → Controller → Browser → DOM
• Pluggable registry: easy to add custom actions or integrate new endpoints
• Feedback loop: after each action, update state and re-invoke the LLM for planning
• Extensibility: custom-functions folder shows how to register user-provided logic
Technology highlights
• Python 3.11+, async/await for concurrency
• Playwright for browser automation
• buildDomTree.js for front-end DOM snapshot
• LangChain (optional) wrappers for LLMs
• Pytest for tests, pre-commit, GitHub Actions for CI/CD
Final checklist
• An inset showing how a developer would add a new action to the registry
• Keep the central LLM-agent-browser loop visually prominent