How emergenceai/agent-e works
Project type and purpose
• Agent-E is a Python-based multi-agent automation tool for driving web browsers via natural-language commands.
• It can be run as a CLI/desktop overlay (injectOverlay.js + UI Manager), or as an HTTP API (FastAPI + Uvicorn).
• At its core it leverages the AutoGen (AG2) framework to coordinate two agents—a high-level planner and a browser-navigation executor—via reusable “skills.”
High-level file-structure analysis
• ae/
– core/agents: planner and browser_nav agent modules
– core/skills: atomic web-action “skills” (click, open_url, get_dom, etc.)
– core/playwright_manager: wraps Playwright browser control
– core/system_orchestrator.py: orchestrates LLM calls, agents, and skill execution
– ui_manager.py + ui/injectOverlay.js: browser-side chat overlay
– server/api_routes.py: FastAPI HTTP endpoints
– main.py: CLI and orchestrator entry point
– utils/: LLM wrappers (OpenAI, Anthropic, Gemini), logging, parsing, etc.
– memory/static_ltm.py: simple file-based long-term memory
• scripts/: support utilities (test conversion, result aggregation)
• test/: end-to-end scenario definitions and runners (webarena-based)
Core components
a. Presentation layer
– Chat UI overlay (injectOverlay.js) ↔ UI Manager (in Python)
– CLI entry (python -m ae.main)
– HTTP API server (FastAPI + Uvicorn → api_routes.py)
b. Orchestration layer
– System Orchestrator: receives user command, loads LLM configs, dispatches to agents
c. Agents
– Planner Agent (high_level_planner_agent.py): maps natural language to a sequence of skill calls
– Browser-Nav Agent (browser_nav_agent.py): executes skills in browser context
d. Skills library
– Sensing Skills (get_dom, get_url, etc.)
– Action Skills (click, enter_text, open_url, pdf_text_extractor, etc.)
– Skill Registry: dynamic loading via ADDITIONAL_SKILL_DIRS
e. Infrastructure & external services
– Playwright Manager: launches & controls headless or real Chrome
– LLM providers: OpenAI / AzureOpenAI / Anthropic / Gemini (via utils/*_llm_helper)
– Static Memory store (user_preferences.txt + static_ltm)
– Optional local vector DB (future, not in code tree)
f. Support utilities
– Notification Manager (for in-browser prompts)
– Post-processing responses, logging, formatters
Relationships & data flows
• User input (CLI / HTTP / UI) → System Orchestrator
• Orchestrator → Planner Agent (LLM call) → returns plan (list of skills)
• Orchestrator → Browser-Nav Agent (LLM call + skill loop)
• Browser-Nav Agent → Playwright Manager → browser actions → returns outcome text
• Outcomes → Orchestrator → UI Manager or HTTP response → User
• Memory reads/writes occur when Planner or Orchestrator requests user preferences or LTM
• Logging & notifications interwoven via Notification Manager & logger
Architectural patterns & design principles
• Layered architecture: Presentation / Orchestration / Agents / Skills / Infrastructure
• Plugin pattern: Skills are self-contained modules registered at runtime
• Separation of concerns:
– High-level planning vs. low-level browser execution
– UI vs. core logic vs. external service adapters
• LLM calls encapsulated in helper classes to isolate API changes
• Extensible memory interface (static now, pluggable for vector DB later)
– User Interfaces (CLI, Browser Overlay, HTTP API)
– Orchestration Engine
– Agents (Planner, Nav)
– Skills Library
– Browser driver and LLM adapters