How wix-incubator/pilot works
Main Components
• Core Library (packages/core):
• Contains the main Pilot class, which exposes APIs like start, perform, autopilot, and end.
• Implements features such as natural language test interpretation, error handling (e.g., PilotError and CodeEvaluationError), and extension of the API catalog.
• Includes utilities, common modules (e.g., APIFormatter, CodeEvaluator, logger, snapshot management), and test utilities.
• Driver Modules (packages/drivers):
• Each subfolder (e.g., detox, puppeteer, playwright, appium, basewebdriver, web-utils) represents an integration point with a specific testing framework.
• Each driver wraps and adapts the native testing APIs to work with the Pilot core.
• They typically include configuration files (e.g., tsconfig.json), examples, and driver-specific utilities (like promptHandler wrappers or custom DOM interrogation functions).
• LLM Handler / Prompt Layer:
• Although not a file per se, the README highlights that integration with an LLM service (like OpenAI or Anthropic) is fundamental.
• The PromptHandler interface and its implementations (such as CustomPromptHandler) are responsible for converting plain text commands into executable steps.
• Website / Documentation:
• The website folder holds the Docusaurus-based documentation. It isn’t part of the runtime system but is important for understanding usage and guiding enhancements.
Relationships and Interactions
• The Pilot instance is created in a client’s test file (see Quick Start examples).
• It depends on one of the driver modules (e.g., DetoxFrameworkDriver, PlaywrightFrameworkDriver) which translates test steps into framework-specific commands.
• It also requires a PromptHandler implementation to send and retrieve intelligent completions from an LLM service.
• Data/Control Flow:
• When a test is initiated, the Pilot class calls start() to initialize a session.
• For each test step, the perform method is called with human-readable commands.
• The Pilot class uses the PromptHandler (LLM) to convert these commands into actionable test operations.
• The selected driver interprets and executes these operations against the target application (web or mobile).
• Error Handling & Logging:
• The core module includes structured error handling and logging, which should be represented as auxiliary channels for reliability and debugging.
Important Architectural Patterns and Design Principles
• Plugin / Modular Driver Architecture:
• The drivers are implemented as separate modules (a clear indication of a plugin pattern) that can be easily extended or replaced.
• Separation of Concerns:
• The core (test orchestration and LLM interface) is entirely separated from platform-specific driver implementations.
• Abstraction via Interfaces:
• The use of interfaces like PromptHandler and TestingFrameworkDriver ensures loose coupling between components.
• Extensibility:
• The ability to extend the API catalog indicates a design aimed at customers who may want to add custom test commands or support for additional frameworks.
Relevant Technologies, Frameworks, and Tools
• Node.js and TypeScript (evident in tsconfig.json and package.json files)
• Various test frameworks: Detox (for mobile), Puppeteer and Playwright (for web), Appium (for mobile automation via WebDriverIO)
• GitHub Actions workflows for continuous integration (linting, testing, deployment)
• Docusaurus for the project website and documentation
Instructing the Principal Software Engineer to Include These Elements
• Data flow from user-written test steps → Pilot Core → PromptHandler → LLM Service.
• Execution flow from Pilot Core → respective Driver → target testing framework (web/mobile).
• Feedback paths, such as error logging and reporting mechanisms.
• Notes on configuration:
• How configuration (via package.json files, tsconfig.json) facilitates the integration and extension mechanism.