How loadmill/node-playwright-cua works
Component Identification
• User Input Interface: Represents commands coming either interactively from the terminal or from an instructions file.
• Main Controller (index.js): Acts as the central loop that reads user input, sends instructions, and processes the API response.
• API Communication Module (openai.js): Builds and sends requests with user commands and browser screenshots to the OpenAI computer-use API while handling conversation context.
• Action Handlers (actions.js): Contains functions to perform browser actions such as click, type, scroll, etc., based on the API’s response.
• Browser Automation Module (browser.js): Launches and manages the Chromium browser using Playwright, including optional HAR file generation.
Defining Relationships and Data Flow
• Loop where after an action is performed by the Action Handlers (which use browser.js), a screenshot is taken and sent back to the API Communication Module to continue the cycle.
• Feedback paths: API responses are processed by the Main Controller which then triggers appropriate browser actions.
Architectural Patterns
• Separation of concerns: one module handles user input and orchestration (index.js), one handles API communications (openai.js), one handles browser interactions (browser.js), and another executes concrete actions (actions.js).
Technologies and Key Details
• That Playwright is specifically used within browser.js.
• That OpenAI’s API is central to decision making and contextual conversation handling.
• That the system supports both interactive and file-based command inputs.