How test-zeus-ai/testzeus-hercules works
─────────────────────────────
PROJECT TYPE & PURPOSE
─────────────────────────────
• The project is an open-source, AI-driven testing agent called “Hercules.”
• Its purpose is to simplify end-to-end test automation by converting Gherkin-based test cases into fully automated web interactions and validations.
• It integrates with LLMs (like GPT-4o and others) using a multi-agent architecture (Planner Agent and Browser Navigation Agent) and a robust tools library to execute web testing tasks, including API, security, accessibility, and visual validations.
─────────────────────────────
ANALYZING THE FILE STRUCTURE
─────────────────────────────
Review the top-level directories and files:
• Directories like “frontend,” “docs,” “helper_scripts,” “tests,” and “testzeus_hercules” hint at numerous facets:
• “frontend”: Contains both interactive and non-interactive UIs, possibly for demonstration or debugging purposes.
• “testzeus_hercules”: The core engine where agents, tools, memory management, and orchestration reside.
• “helper_scripts”: A collection of scripts to manage execution, setup, and additional tooling (like connecting to remote browser instances via CDP URLs).
• “tests”: Provides example Gherkin feature files and supporting test data, showing how automated tests are executed.
• Dockerfile, .env-example, and GitHub workflows underscore containerization and CI/CD integration.
• Notice configuration files (pyproject.toml, Makefile, environment docs), which indicate that the system’s behavior is heavily parameterized via environment variables and configuration management.
─────────────────────────────
EXTRACTING ARCHITECTURAL INSIGHTS FROM THE README
─────────────────────────────
The README elaborates on:
• How users provide input via Gherkin files, configure test parameters, and set environment variables.
• The multiple approaches to running Hercules (via PyPI package, Docker container, source code build, and helper scripts).
• A detailed explanation of the multi-agent architecture:
• “Planner Agent” decomposes tasks by interpreting Gherkin steps.
• “Browser Navigation Agent” leverages tools to perform actions (clicks, text entry, validations, etc.) via a dedicated Playwright manager.
• An emphasis on modularity in the Tools Library with “sensing tools” (to capture the DOM, current URL) and “action tools” (to interact with the browser).
• Integration with external services (like BrowserStack, LambdaTest, BrowserBase) and the setup for connecting to remote browser instances using CDP endpoints.
• Mechanisms for logging, output reporting (HTML and JUnit XML reports), and capturing execution proofs (screenshots, videos, network logs).
─────────────────────────────
─────────────────────────────
A. MAIN COMPONENTS & SUBSYSTEMS
a. User Interface / CLI Layer
• Represents how a user triggers tests (e.g., by running the “testzeus-hercules” command).
• Can also include interactive modes via the “frontend” folder where a browser-based UI is provided.
b. Input Components
• Gherkin Feature Files: Shown as external input files (located under “opt/input” and “tests”) that are fed into Hercules.
• Test Data and Configurations: Represented by the “opt/test_data” folder, environment files (.env), and agents_llm_config.json.
c. Core Engine/Orchestration
• Planner Agent: Acts as the decision-making component that breaks down each Gherkin step into actionable tasks.
• Browser Navigation Agent: Interacts with the browser for tests, using Playwright as a bridge.
• Memory Modules: Both dynamic (for session context) and static (pre-loaded data), located within testzeus_hercules/core/memory.
d. Tools Library
• Sensing Tools: Components like geturl, get_dom_with_content_type, etc.
• Action Tools: Components like click, enter_text, openurl, etc.
• Additional Tools: Located in “additional_tools” and integrated via configuration (through ADDITIONAL_TOOL_DIRS).
e. External Integration Subsystem
• Browser Services: Connections via Playwright and optional remote browser endpoints (e.g., CDP_ENDPOINT_URL for BrowserBase or BrowserStack).
• CI/CD & Docker Integration: Show the Docker container execution mode with volume mounts, environment files, and automation scripts.
f. Logging & Reporting
• Output Path: Directory structure for logs, HTML reports, JUnit XML files, and captured proofs (videos, screenshots, network logs).
• Chat Logs: Represent “chat_messages.json” in the log_files folder which captures the chain of thought for diagnostic purposes.
B. RELATIONSHIPS & DATA FLOWS
• Feedback loops where execution results (logs, proofs) are sent back to the core engine and eventually output to reporting directories.
C. ARCHITECTURAL PATTERNS & DESIGN PRINCIPLES
• Modular, multi-agentic architecture, where each agent and tool is clearly isolated and loosely coupled yet coordinated by the core runner.
• Separation between test planning and test execution, ensuring a clean division of responsibilities.
• Use of environment-driven configurations to allow dynamic behavior and cross-environment compatibility, especially for Docker/CICD environments.
D. TECHNOLOGIES & FRAMEWORKS
• Python 3.11: Foundational language version.
• Playwright: Used for browser automation across testing scenarios.
• Docker: For containerized testing deployments.
• LLMs: External APIs (OpenAI GPT-4o, Anthropic, etc.) for intelligent decision-making.
• CI/CD: GitHub workflows and Makefile for automation.
─────────────────────────────
FINAL REMINDER
─────────────────────────────