How red-sat/webact works
Main Components
• Agent Component: The core is the WebActAgent (implemented in webact/agent.py) which coordinates action prediction, browser navigation, state management, and interaction with various helper utilities.
• Utility and Helper Modules:
• Data Utilities (data_utils folder): Functions dedicated to prompt formatting and engineering.
• Demo Utilities (demo_utils folder): Components responsible for browser control (browser_helper.py), crawling (crawler_helper.py), prompt formatting (format_prompt.py), and model inference (inference_engine.py).
• External Tools/Services:
• Web browser automation via Playwright.
• Integration with external LLM services (e.g., Gemini 2.0, GPT-4o, LLaVA) as described in the README.
• Configuration/Testing Tools:
• quick_test.py for operational demonstrations.
• requirements.txt as dependency management.
Establish Relationships and Interactions
• The WebActAgent drives the overall workflow: it initiates and orchestrates operations by calling utility methods to generate and format prompts (data_utils) and execute automation tasks via browser helper and crawler (demo_utils).
• The Inference Engine in demo_utils/inference_engine.py interacts with the external AI models, effectively bridging between the agent’s prompt requests and receiving predictions.
• Data flows from the agent to the inference engine (for predictions), from the agent to the browser helper/crawler (for actual web operations), and back to the agent for decision-making and iteration.
• Configuration parameters, such as API keys and agent parameters (max_auto_op, grounding_strategy etc.), influence component behavior.
Architectural Patterns and Design Principles
• Modularity: Each module (agent, prompt formatting, browser control, crawling, inference) is separated into distinct directories to support a plug-and-play approach.
• Separation of Concerns: The project clearly demarcates the AI integration elements, the data transformation utilities, and the web navigation subprocess.
• Extensibility: Switching between different models (Gemini, GPT, LLaVA) and configuring tasks via parameters reveal a design ready for future expansion.
• Asynchronous operation: The provided usage example indicates asynchronous agent behavior in managing predictions and actions.
Relevant Technologies and Frameworks
• Python as the primary environment with dependencies managed through requirements.txt.
• Playwright for browser automation.
• Large Language Models via external API integrations (e.g., Gemini and GPT).
• Asynchronous programming (e.g., using async/await within agent operations).