# osu-nlp-group/seeact architecture

> Architecture diagram of the GitHub repository osu-nlp-group/seeact, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/osu-nlp-group/seeact
- Repository: https://github.com/osu-nlp-group/seeact

## Overview

##### Main Components Identification

a. Web Agent Core (SeeActAgent)
- This is the central engine that drives the agent’s behavior, communicates with the LMM (e.g., GPT-4V(ision), GPT-4-turbo, etc.), and manages task execution.
- Located in seeact_package/seeact/agent.py and also used by src/seeact.py as the entry point.

b. Browser Integration Layer
- Handled by Playwright and supported by demo utilities (seeact/demo_utils and src/demo_utils). These modules (e.g., browser_helper.py, crawler_helper.py) abstract browser interactions (tunneling inputs, executing predicted actions, handling crawler mode).

c. Data Utilities and Formatting
- Several modules under data_utils (both in seeact_package/seeact/data_utils and src/data_utils) are responsible for formatting prompts and handling DOM evaluation, image processing, and evaluation utilities.

d. Configuration and Mode Management
- The project uses TOML configuration files (src/config/*.toml) that define parameters for demo mode, auto mode, online experiments, and offline experiments.
- These configurations manage default tasks, website URLs, maximum operation counts, output directories, and other settings.

e. Offline Experiments and Data Processing
- Under src/offline_experiments, there are scripts for screenshot generation and annotation (element_attributes.py, image_annotation.py, textual_choices.py) that prepare data for evaluation.
- The data directory holds examples and online task JSONs used for batch experiments.

f. External Dependencies & APIs
- The system interacts with external LLM APIs (e.g., OpenAI, Google, or Ollama) for processing tasks. API key handling is clearly described in the README.
- It integrates with third-party libraries such as Playwright for browser automation.

##### Relationships and Data Flow

- User or configuration input provides tasks and websites via either the command-line (demo mode) or JSON files (auto mode).
- These inputs are fed into the SeeActAgent.
- The SeeActAgent then sends prompt data (and formatted content via data_utils) to LLM APIs, receives prediction dictionaries, and interacts with the browser using the demo_utils layer.
- Browser events (clicks, typing, selections) are executed on live websites. There might be feedback loops where the agent monitors browser responses.
- Offline experiments and data pipelines use similar modules but operate over datasets found in the data/ directory (e.g., Mind2Web examples) to benchmark or generate screenshot/query data.
- Configuration files in src/config set the parameters for each mode, and these govern the behavior of the agent.

##### Architectural Patterns and Design Principles

- Separation of Concerns: The system is modularized into distinct components (agent logic, browser interfacing, data formatting, and experiments). Each module/file tree in seeact_package and src represents a separate layer.
- API Abstraction: The SeeActAgent abstracts the call to LLM APIs, allowing the same high-level logic to be used with different models.
- Configurability: Use of external TOML files allows dynamic reconfiguration for demo, auto, and online modes.
- Data-Driven Operation: The system leverages structured JSON and annotated data (the Mind2Web dataset) to automate and evaluate tasks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        UI["User Input"]:::inputLayer
        Config["Configuration (src/config/*.toml)"]:::inputLayer
    end

    %% Core Layer
    subgraph "Core Layer"
        Agent["Agent (seeact_package/seeact/agent.py, src/seeact.py)"]:::agentCore
        LLM["External LLM APIs (OpenAI,Google,Ollama)"]:::agentCore
        DataUtils["Data Utilities (seeact/data_utils & src/data_utils)"]:::agentCore
    end

    %% Integration Layer
    subgraph "Integration Layer"
        BrowserIntegration["Browser Integration (Demo Utils: browser_helper,crawler_helper,format_prompt)"]:::integration
    end

    %% Offline Processing Layer
    subgraph "Offline Processing Layer"
        OfflineExp["Offline Experiments (src/offline_experiments)"]:::offline
        DataRepo["Data Repository (data/examples, data/online_tasks)"]:::offline
    end

    %% Data Flow Connections
    UI -->|"TaskInput"| Agent
    Config -->|"ModeSettings"| Agent
    Agent -->|"APICall"| LLM
    LLM -->|"Prediction"| Agent
    Agent -->|"DataFormatting"| DataUtils
    DataUtils -->|"FormattedData"| Agent
    Agent -->|"BrowserEventExecution"| BrowserIntegration
    BrowserIntegration -->|"BrowserFeedback"| Agent
    DataRepo -->|"DataProcessing"| OfflineExp

    %% Click Events for Web Agent Core
    click Agent "https://github.com/osu-nlp-group/seeact/blob/main/seeact_package/seeact/agent.py"
    click Agent "https://github.com/osu-nlp-group/seeact/blob/main/src/seeact.py"

    %% Click Events for Browser Integration Layer
    click BrowserIntegration "https://github.com/osu-nlp-group/seeact/blob/main/seeact_package/seeact/demo_utils/browser_helper.py"
    click BrowserIntegration "https://github.com/osu-nlp-group/seeact/blob/main/seeact_package/seeact/demo_utils/crawler_helper.py"
    click BrowserIntegration "https://github.com/osu-nlp-group/seeact/blob/main/src/demo_utils/browser_helper.py"
    click BrowserIntegration "https://github.com/osu-nlp-group/seeact/blob/main/src/demo_utils/format_prompt.py"

    %% Click Events for Data Utilities and Formatting
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/seeact_package/seeact/data_utils/format_prompt_utils.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/seeact_package/seeact/data_utils/prompts.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/src/data_utils/dom_utils.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/src/data_utils/evaluation_utils.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/src/data_utils/format_prompt_utils.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/src/data_utils/image_utils.py"
    click DataUtils "https://github.com/osu-nlp-group/seeact/blob/main/src/data_utils/prompts.py"

    %% Click Events for Configuration and Mode Management
    click Config "https://github.com/osu-nlp-group/seeact/blob/main/src/config/auto_mode.toml"
    click Config "https://github.com/osu-nlp-group/seeact/blob/main/src/config/demo_mode.toml"
    click Config "https://github.com/osu-nlp-group/seeact/blob/main/src/config/online_exp.toml"

    %% Click Events for Offline Experiments and Data Processing
    click OfflineExp "https://github.com/osu-nlp-group/seeact/blob/main/src/offline_experiments/offline_experiment.py"
    click OfflineExp "https://github.com/osu-nlp-group/seeact/blob/main/src/offline_experiments/screenshot_generation/element_attributes.py"
    click OfflineExp "https://github.com/osu-nlp-group/seeact/blob/main/src/offline_experiments/screenshot_generation/image_annotation.py"
    click OfflineExp "https://github.com/osu-nlp-group/seeact/blob/main/src/offline_experiments/screenshot_generation/textual_choices.py"

    %% Click Events for Data Repository / Datasets
    click DataRepo "https://github.com/osu-nlp-group/seeact/tree/main/data/examples/"
    click DataRepo "https://github.com/osu-nlp-group/seeact/tree/main/data/online_tasks/"

    %% Styling
    classDef inputLayer fill:#BBDEFB,stroke:#1E88E5,stroke-width:2px;
    classDef agentCore fill:#FFECB3,stroke:#F57C00,stroke-width:2px;
    classDef integration fill:#C8E6C9,stroke:#388E3C,stroke-width:2px;
    classDef offline fill:#F8BBD0,stroke:#C2185B,stroke-width:2px;
```
