# openai/openai-testing-agent-demo architecture

> Architecture diagram of the GitHub repository openai/openai-testing-agent-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/openai-testing-agent-demo
- Repository: https://github.com/openai/openai-testing-agent-demo

## Overview

##### Project Type and Purpose
- This is a full-stack monorepo that demonstrates an AI-driven automated frontend testing solution.
- It contains three cooperating applications:
a. frontend – A Next.js UI for configuring tests and viewing results.
b. cua-server – A Node.js microservice that orchestrates the OpenAI CUA model and Playwright-driven browser actions.
c. sample-test-app – A toy e-commerce Next.js site under test.

##### File Structure Analysis
- Top-level folders:
- frontend/ – UI code (app/, components/, stores/, lib/)
- cua-server/ – Backend service (src/handlers/, src/agents/, src/services/, src/lib/)
- sample-test-app/ – Example target application (app/api/, components/, lib/, stores/)
- Monorepo root contains package.json for orchestrating dev commands.
- Each app has its own README, .env.example, tsconfig.json, and build configs.
- No persistent database; state is in the sample-app stores or in-memory within the Node service.

##### README Insights
- The core testing logic lives in cua-server and leverages:
- OpenAI’s CUA (computer use) model via openai-cua-client
- Playwright to launch and control a headless browser against sample-test-app
- Socket.IO/WebSockets for real-time updates between frontend and cua-server
- Ports:
- Frontend at localhost:3000
- Sample-test-app at localhost:3005
- CUA server WebSocket at ws://localhost:8080

##### Main Components
a. Frontend UI (Next.js @3000)
- ConfigPanel, SidePanel, TaskSteps components
- SocketIOManager interacts via WebSocket
b. CUA Server (Node.js @8080)
- Handlers (test-case initiation, update, user messages)
- Agents (test-case-agent, test-script-review-agent)
- Services (login, openai-cua-client)
- Lib modules (computer-use-loop orchestrator)
c. Sample Test App (Next.js @3005)
- API routes (auth, orders, store)
- UI components (CartView, ItemCard, etc.)
d. External Service
- OpenAI API (CUA model endpoints)
e. Playwright Browser Instance
- Launched and controlled by cua-server to navigate sample-test-app

##### Relationships and Data Flows
1. User → Browser: open http://localhost:3000
2. Frontend UI → CUA Server over WebSocket: “Start Test” command with test case config
3. CUA Server → OpenAI API: request next action via CUA model
4. CUA Server → Playwright: translate model response into browser actions on sample-test-app
5. Playwright → Sample-Test-App (HTTP): perform clicks, inputs, navigation
6. CUA Server → Frontend UI over WebSocket: stream logs, state updates, screenshots

##### Architectural Patterns and Principles
- Microservice separation: distinct apps for UI, orchestration, and target under test
- Event-driven loop: cua-loop-handler drives a continual request/response cycle with the model
- Socket-based real-time updates between backend and frontend
- Modular “agent” architecture within cua-server for different test-case tasks
- Clear separation of concerns:
- UI layer (React/Next.js)
- Orchestration layer (Node.js with handlers, services, agents)
- Target application under test (Next.js sample app)

##### Technologies and Frameworks
- Frontend: Next.js 13+, React, Tailwind CSS, Socket.IO, Zustand (stores)
- CUA Server: Node.js, TypeScript, Socket.IO, OpenAI SDK, Playwright
- Sample App: Next.js 13+, React, API Routes, CSV data import
- Monorepo: npm workspaces, tsconfig paths

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Layer"
        Browser(["🧑‍💻 Browser<br/>(port3000)"]):::user
    end

    subgraph "Client Layer"
        FrontendUI["Next.js Frontend<br/>(3000)"]:::frontend
        subgraph "FE Components"
            ConfigPanel["ConfigPanel"]:::frontend
            SidePanel["SidePanel"]:::frontend
            TaskSteps["TaskSteps"]:::frontend
            SocketIOManager["SocketIOManager"]:::frontend
            MainPageFE["page.tsx"]:::frontend
        end
    end

    subgraph "Orchestration Layer"
        CUAServer["CUA Server<br/>(WS 8080)"]:::orchestration
        subgraph "Handlers"
            HandlerInit["test-case-initiation-handler"]:::orchestration
            HandlerUpdate["test-case-update-handler"]:::orchestration
            HandlerUserMsgs["user-messages-handler"]:::orchestration
            HandlerLoop["cua-loop-handler"]:::orchestration
            HandlerAction["action-handler"]:::orchestration
        end
        subgraph "Agents"
            AgentTC["test-case-agent"]:::orchestration
            AgentReview["test-script-review-agent"]:::orchestration
        end
        subgraph "Services"
            ServiceOpenAI["openai-cua-client"]:::orchestration
            ServiceLogin["login-service"]:::orchestration
        end
        LibLoop["computer-use-loop"]:::orchestration
        IndexServer["index.ts"]:::orchestration
        Playwright["🎭 Headless Browser"]:::playwright
    end

    subgraph "Target Layer"
        SampleApp["Next.js Sample App<br/>(3005)"]:::target
        subgraph "API Routes"
            APIAuth["auth/login API"]:::target
            APIOrders["orders API"]:::target
            APIStyleID["store/styles/[id] API"]:::target
            APIStyle["store/styles API"]:::target
        end
        subgraph "UI Components"
            CartView["CartView"]:::target
            ItemCard["ItemCard"]:::target
            ShopContent["ShopPageContent"]:::target
            OrderSuccess["OrderSuccess"]:::target
        end
        LayoutTA["layout.tsx"]:::target
        PageTA["page.tsx"]:::target
    end

    subgraph "External"
        OpenAI["OpenAI API"]:::external
    end

    Browser -->|"HTTP GET /"| FrontendUI
    FrontendUI -->|"WebSocket: Start Test"| CUAServer
    CUAServer -->|"OpenAI API call"| OpenAI
    CUAServer -->|"Playwright commands"| Playwright
    Playwright -->|"HTTP actions"| SampleApp
    SampleApp -->|"HTTP responses"| Playwright
    CUAServer -->|"WebSocket events"| FrontendUI

    click FrontendUI "https://github.com/openai/openai-testing-agent-demo/tree/main/frontend/"
    click ConfigPanel "https://github.com/openai/openai-testing-agent-demo/blob/main/frontend/components/ConfigPanel.tsx"
    click SidePanel "https://github.com/openai/openai-testing-agent-demo/blob/main/frontend/components/SidePanel.tsx"
    click TaskSteps "https://github.com/openai/openai-testing-agent-demo/blob/main/frontend/components/TaskSteps.tsx"
    click SocketIOManager "https://github.com/openai/openai-testing-agent-demo/blob/main/frontend/components/SocketIOManager.tsx"
    click MainPageFE "https://github.com/openai/openai-testing-agent-demo/blob/main/frontend/app/page.tsx"

    click CUAServer "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/index.ts"
    click HandlerInit "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/handlers/test-case-initiation-handler.ts"
    click HandlerUpdate "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/handlers/test-case-update-handler.ts"
    click HandlerUserMsgs "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/handlers/user-messages-handler.ts"
    click HandlerLoop "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/handlers/cua-loop-handler.ts"
    click HandlerAction "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/handlers/action-handler.ts"
    click AgentTC "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/agents/test-case-agent.ts"
    click AgentReview "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/agents/test-script-review-agent.ts"
    click ServiceOpenAI "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/services/openai-cua-client.ts"
    click ServiceLogin "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/services/login-service.ts"
    click LibLoop "https://github.com/openai/openai-testing-agent-demo/blob/main/cua-server/src/lib/computer-use-loop.ts"

    click SampleApp "https://github.com/openai/openai-testing-agent-demo/tree/main/sample-test-app/"
    click APIAuth "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/api/auth/login/route.ts"
    click APIOrders "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/api/services/orders/route.ts"
    click APIStyleID "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/api/store/styles/[id]/route.ts"
    click APIStyle "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/api/store/styles/route.ts"
    click CartView "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/components/CartView.tsx"
    click ItemCard "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/components/ItemCard.tsx"
    click ShopContent "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/components/ShopPageContent.tsx"
    click OrderSuccess "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/components/OrderSuccess.tsx"
    click LayoutTA "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/layout.tsx"
    click PageTA "https://github.com/openai/openai-testing-agent-demo/blob/main/sample-test-app/app/page.tsx"

    classDef user fill:#FFEB3B,stroke:#FBC02D,color:#000
    classDef frontend fill:#BBDEFB,stroke:#2196F3,color:#000
    classDef orchestration fill:#C8E6C9,stroke:#4CAF50,color:#000
    classDef playwright fill:#FFE0B2,stroke:#FF9800,color:#000
    classDef target fill:#D1C4E9,stroke:#7E57C2,color:#000
    classDef external fill:#C8E6C9,stroke:#388E3C,color:#000
```
