How openai/openai-testing-agent-demo works
Project Type and Purpose
• This is a full-stack monorepo that demonstrates an AI-driven automated frontend testing solution.
• It contains three cooperating applications:
a. frontend – A Next.js UI for configuring tests and viewing results.
b. cua-server – A Node.js microservice that orchestrates the OpenAI CUA model and Playwright-driven browser actions.
c. sample-test-app – A toy e-commerce Next.js site under test.
File Structure Analysis
• Top-level folders:
• frontend/ – UI code (app/, components/, stores/, lib/)
• cua-server/ – Backend service (src/handlers/, src/agents/, src/services/, src/lib/)
• sample-test-app/ – Example target application (app/api/, components/, lib/, stores/)
• Monorepo root contains package.json for orchestrating dev commands.
• Each app has its own README, .env.example, tsconfig.json, and build configs.
• No persistent database; state is in the sample-app stores or in-memory within the Node service.
README Insights
• The core testing logic lives in cua-server and leverages:
• OpenAI’s CUA (computer use) model via openai-cua-client
• Playwright to launch and control a headless browser against sample-test-app
• Socket.IO/WebSockets for real-time updates between frontend and cua-server
• Ports:
• Frontend at localhost:3000
• Sample-test-app at localhost:3005
• CUA server WebSocket at ws://localhost:8080
Main Components
a. Frontend UI (Next.js @3000)
• ConfigPanel, SidePanel, TaskSteps components
• SocketIOManager interacts via WebSocket
b. CUA Server (Node.js @8080)
• Handlers (test-case initiation, update, user messages)
• Agents (test-case-agent, test-script-review-agent)
• Services (login, openai-cua-client)
• Lib modules (computer-use-loop orchestrator)
c. Sample Test App (Next.js @3005)
• API routes (auth, orders, store)
• UI components (CartView, ItemCard, etc.)
d. External Service
• OpenAI API (CUA model endpoints)
e. Playwright Browser Instance
• Launched and controlled by cua-server to navigate sample-test-app
Relationships and Data Flows
1. User → Browser: open http://localhost:3000
2. Frontend UI → CUA Server over WebSocket: “Start Test” command with test case config
3. CUA Server → OpenAI API: request next action via CUA model
4. CUA Server → Playwright: translate model response into browser actions on sample-test-app
5. Playwright → Sample-Test-App (HTTP): perform clicks, inputs, navigation
6. CUA Server → Frontend UI over WebSocket: stream logs, state updates, screenshots
Architectural Patterns and Principles
• Microservice separation: distinct apps for UI, orchestration, and target under test
• Event-driven loop: cua-loop-handler drives a continual request/response cycle with the model
• Socket-based real-time updates between backend and frontend
• Modular “agent” architecture within cua-server for different test-case tasks
• Clear separation of concerns:
• UI layer (React/Next.js)
• Orchestration layer (Node.js with handlers, services, agents)
• Target application under test (Next.js sample app)
Technologies and Frameworks
• Frontend: Next.js 13+, React, Tailwind CSS, Socket.IO, Zustand (stores)
• CUA Server: Node.js, TypeScript, Socket.IO, OpenAI SDK, Playwright
• Sample App: Next.js 13+, React, API Routes, CSV data import
• Monorepo: npm workspaces, tsconfig paths