# emergenceai/agent-e architecture

> Architecture diagram of the GitHub repository emergenceai/agent-e, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emergenceai/agent-e
- Repository: https://github.com/emergenceai/agent-e

## Overview

##### Project type and purpose
- Agent-E is a Python-based multi-agent automation tool for driving web browsers via natural-language commands.
- It can be run as a CLI/desktop overlay (injectOverlay.js + UI Manager), or as an HTTP API (FastAPI + Uvicorn).
- At its core it leverages the AutoGen (AG2) framework to coordinate two agents—a high-level planner and a browser-navigation executor—via reusable “skills.”

##### High-level file-structure analysis
- ae/
– core/agents: planner and browser_nav agent modules
– core/skills: atomic web-action “skills” (click, open_url, get_dom, etc.)
– core/playwright_manager: wraps Playwright browser control
– core/system_orchestrator.py: orchestrates LLM calls, agents, and skill execution
– ui_manager.py + ui/injectOverlay.js: browser-side chat overlay
– server/api_routes.py: FastAPI HTTP endpoints
– main.py: CLI and orchestrator entry point
– utils/: LLM wrappers (OpenAI, Anthropic, Gemini), logging, parsing, etc.
– memory/static_ltm.py: simple file-based long-term memory
- scripts/: support utilities (test conversion, result aggregation)
- test/: end-to-end scenario definitions and runners (webarena-based)

##### Core components
a. Presentation layer
– Chat UI overlay (injectOverlay.js) ↔ UI Manager (in Python)
– CLI entry (python -m ae.main)
– HTTP API server (FastAPI + Uvicorn → api_routes.py)
b. Orchestration layer
– System Orchestrator: receives user command, loads LLM configs, dispatches to agents
c. Agents
– Planner Agent (high_level_planner_agent.py): maps natural language to a sequence of skill calls
– Browser-Nav Agent (browser_nav_agent.py): executes skills in browser context
d. Skills library
– Sensing Skills (get_dom, get_url, etc.)
– Action Skills (click, enter_text, open_url, pdf_text_extractor, etc.)
– Skill Registry: dynamic loading via ADDITIONAL_SKILL_DIRS
e. Infrastructure & external services
– Playwright Manager: launches & controls headless or real Chrome
– LLM providers: OpenAI / AzureOpenAI / Anthropic / Gemini (via utils/*_llm_helper)
– Static Memory store (user_preferences.txt + static_ltm)
– Optional local vector DB (future, not in code tree)
f. Support utilities
– Notification Manager (for in-browser prompts)
– Post-processing responses, logging, formatters

##### Relationships & data flows
- User input (CLI / HTTP / UI) → System Orchestrator
- Orchestrator → Planner Agent (LLM call) → returns plan (list of skills)
- Orchestrator → Browser-Nav Agent (LLM call + skill loop)
- Browser-Nav Agent → Playwright Manager → browser actions → returns outcome text
- Outcomes → Orchestrator → UI Manager or HTTP response → User
- Memory reads/writes occur when Planner or Orchestrator requests user preferences or LTM
- Logging & notifications interwoven via Notification Manager & logger

##### Architectural patterns & design principles
- Layered architecture: Presentation / Orchestration / Agents / Skills / Infrastructure
- Plugin pattern: Skills are self-contained modules registered at runtime
- Separation of concerns:
– High-level planning vs. low-level browser execution
– UI vs. core logic vs. external service adapters
- LLM calls encapsulated in helper classes to isolate API changes
- Extensible memory interface (static now, pluggable for vector DB later)

– User Interfaces (CLI, Browser Overlay, HTTP API)
– Orchestration Engine
– Agents (Planner, Nav)
– Skills Library
– Browser driver and LLM adapters

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        CLI["CLI Entry Point<br/>(ae/main.py)"]:::presentation
        UIOverlay["Chat UI Overlay"]:::presentation
        UIManager["UI Manager"]:::presentation
        HTTPAPI["HTTP API Server"]:::presentation
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Orchestrator["System Orchestrator"]:::orchestration
    end

    %% Agents Layer
    subgraph "Agents Layer"
        Planner["Planner Agent"]:::agents
        BrowserNav["Browser-Nav Agent"]:::agents
    end

    %% Skills Layer
    subgraph "Skills Layer"
        Sensing["Sensing Skills"]:::skills
        Action["Action Skills"]:::skills
        Registry["Skill Registry"]:::skills
    end

    %% Infrastructure & External Services
    subgraph "Infrastructure & External Services"
        Playwright["Playwright Manager"]:::infra
        OpenAI["OpenAI LLM Adapter"]:::external
        Anthropic["Anthropic LLM Adapter"]:::external
        Gemini["Gemini LLM Adapter"]:::external
        StaticLTM["Static LTM Store"]:::memory
        UserPrefs["User Preferences File"]:::memory
    end

    %% Support Utilities
    subgraph "Support Utilities"
        Notification["Notification Manager"]:::support
        PostProcess["Post-Processing Responses"]:::support
        Logger["Logger Utility"]:::support
        Formatting["Formatting Helper"]:::support
    end

    %% Data Flows
    CLI -->|User Input| Orchestrator
    UIOverlay -->|User Input| UIManager
    UIManager -->|Forwards Command| Orchestrator
    HTTPAPI -->|Receives Request| Orchestrator

    Orchestrator -->|Request Plan| Planner
    Planner -->|Returns Plan| Orchestrator

    Orchestrator -->|Execute Plan| BrowserNav
    BrowserNav -->|Invoke Skills| Registry
    Registry --> Sensing
    Registry --> Action

    BrowserNav -->|Control Browser| Playwright
    Playwright -->|Action Outcome| BrowserNav
    BrowserNav -->|Result| Orchestrator

    Orchestrator -->|Responds To| UIManager
    Orchestrator -->|Responds To| HTTPAPI
    Orchestrator -->|Outputs To| CLI

    Planner --> StaticLTM
    Planner --> UserPrefs
    Orchestrator --> StaticLTM

    Planner --> OpenAI
    Planner --> Anthropic
    Planner --> Gemini
    BrowserNav --> OpenAI
    BrowserNav --> Anthropic
    BrowserNav --> Gemini

    Orchestrator --> Notification
    Orchestrator --> Logger
    Orchestrator --> PostProcess
    Orchestrator --> Formatting

    %% Click Events
    click CLI "https://github.com/emergenceai/agent-e/blob/master/ae/main.py"
    click UIOverlay "https://github.com/emergenceai/agent-e/blob/master/ae/ui/injectOverlay.js"
    click UIManager "https://github.com/emergenceai/agent-e/blob/master/ae/core/ui_manager.py"
    click HTTPAPI "https://github.com/emergenceai/agent-e/blob/master/ae/server/api_routes.py"
    click Orchestrator "https://github.com/emergenceai/agent-e/blob/master/ae/core/system_orchestrator.py"
    click Planner "https://github.com/emergenceai/agent-e/blob/master/ae/core/agents/high_level_planner_agent.py"
    click BrowserNav "https://github.com/emergenceai/agent-e/blob/master/ae/core/agents/browser_nav_agent.py"
    click Sensing "https://github.com/emergenceai/agent-e/tree/master/ae/core/skills"
    click Action "https://github.com/emergenceai/agent-e/tree/master/ae/core/skills"
    click Registry "https://github.com/emergenceai/agent-e/blob/master/ae/core/skills/skill_registry.py"
    click Playwright "https://github.com/emergenceai/agent-e/blob/master/ae/core/playwright_manager.py"
    click OpenAI "https://github.com/emergenceai/agent-e/blob/master/ae/utils/openai_llm_helper.py"
    click Anthropic "https://github.com/emergenceai/agent-e/blob/master/ae/utils/anthropic_llm_helper.py"
    click Gemini "https://github.com/emergenceai/agent-e/blob/master/ae/utils/gemini_llm_helper.py"
    click StaticLTM "https://github.com/emergenceai/agent-e/blob/master/ae/core/memory/static_ltm.py"
    click UserPrefs "https://github.com/emergenceai/agent-e/blob/master/ae/user_preferences/user_preferences.txt"
    click Notification "https://github.com/emergenceai/agent-e/blob/master/ae/core/notification_manager.py"
    click PostProcess "https://github.com/emergenceai/agent-e/blob/master/ae/core/post_process_responses.py"
    click Logger "https://github.com/emergenceai/agent-e/blob/master/ae/utils/logger.py"
    click Formatting "https://github.com/emergenceai/agent-e/blob/master/ae/utils/formatting_helper.py"

    %% Styles
    classDef presentation fill:#cce5ff,stroke:#1e90ff,color:#0b3d91;
    classDef orchestration fill:#d4edda,stroke:#28a745,color:#155724;
    classDef agents fill:#c3e6cb,stroke:#28a745,color:#155724;
    classDef skills fill:#fff3cd,stroke:#ffecb5,color:#856404;
    classDef infra fill:#e2e3e5,stroke:#6c757d,color:#343a40;
    classDef external fill:#e2e3e5,stroke:#6c757d,color:#343a40;
    classDef memory fill:#f5c6cb,stroke:#721c24,color:#721c24;
    classDef support fill:#e2e3ff,stroke:#6f42c1,color:#2e086c;
```
