How osu-nlp-group/seeact works
Main Components Identification
a. Web Agent Core (SeeActAgent)
• This is the central engine that drives the agent’s behavior, communicates with the LMM (e.g., GPT-4V(ision), GPT-4-turbo, etc.), and manages task execution.
• Located in seeact_package/seeact/agent.py and also used by src/seeact.py as the entry point.
b. Browser Integration Layer
• Handled by Playwright and supported by demo utilities (seeact/demo_utils and src/demo_utils). These modules (e.g., browser_helper.py, crawler_helper.py) abstract browser interactions (tunneling inputs, executing predicted actions, handling crawler mode).
c. Data Utilities and Formatting
• Several modules under data_utils (both in seeact_package/seeact/data_utils and src/data_utils) are responsible for formatting prompts and handling DOM evaluation, image processing, and evaluation utilities.
d. Configuration and Mode Management
• The project uses TOML configuration files (src/config/*.toml) that define parameters for demo mode, auto mode, online experiments, and offline experiments.
• These configurations manage default tasks, website URLs, maximum operation counts, output directories, and other settings.
e. Offline Experiments and Data Processing
• Under src/offline_experiments, there are scripts for screenshot generation and annotation (element_attributes.py, image_annotation.py, textual_choices.py) that prepare data for evaluation.
• The data directory holds examples and online task JSONs used for batch experiments.
f. External Dependencies & APIs
• The system interacts with external LLM APIs (e.g., OpenAI, Google, or Ollama) for processing tasks. API key handling is clearly described in the README.
• It integrates with third-party libraries such as Playwright for browser automation.
Relationships and Data Flow
• User or configuration input provides tasks and websites via either the command-line (demo mode) or JSON files (auto mode).
• These inputs are fed into the SeeActAgent.
• The SeeActAgent then sends prompt data (and formatted content via data_utils) to LLM APIs, receives prediction dictionaries, and interacts with the browser using the demo_utils layer.
• Browser events (clicks, typing, selections) are executed on live websites. There might be feedback loops where the agent monitors browser responses.
• Offline experiments and data pipelines use similar modules but operate over datasets found in the data/ directory (e.g., Mind2Web examples) to benchmark or generate screenshot/query data.
• Configuration files in src/config set the parameters for each mode, and these govern the behavior of the agent.
Architectural Patterns and Design Principles
• Separation of Concerns: The system is modularized into distinct components (agent logic, browser interfacing, data formatting, and experiments). Each module/file tree in seeact_package and src represents a separate layer.
• API Abstraction: The SeeActAgent abstracts the call to LLM APIs, allowing the same high-level logic to be used with different models.
• Configurability: Use of external TOML files allows dynamic reconfiguration for demo, auto, and online modes.
• Data-Driven Operation: The system leverages structured JSON and annotated data (the Mind2Web dataset) to automate and evaluate tasks.