bug0inc/passmark

1. Library façade and public API

- src/index.ts is the main entrypoint and re-exports the package surface: configure(), runSteps(), runUserFlow(), assert(), extraction helpers, email providers, and utility types.

- This repo is not an app; it is a TypeScript Playwright testing library that sits inside a user’s Playwright test suite and drives a page object.

2. Primary execution path: natural-language steps over Playwright

- The core subsystem is runSteps() in the src/ execution modules, which converts step descriptions into browser actions against Playwright.

- The data flow is: step request → cache lookup → AI-guided step execution or cached action replay → optional healing/fallback → result returned to the test.

- runUserFlow() is the higher-level variant that treats an entire flow more like a single agentic task rather than a step-by-step script.

3. Cache and state boundary

- src/redis.ts and src/data-cache.ts back Redis-based caching and cross-run data storage.

- Successful single-step actions are cached by flow/step identity and replayed on later runs to reduce latency and cost.

- The same Redis-backed layer also supports {{global.*}} and {{data.*}} placeholder resolution; without Redis, those features degrade or disable cleanly.

4. AI model orchestration

- src/config.ts, src/models.ts, src/constants.ts, and src/tools.ts define the model slots, gateway selection, and the AI tool surface.

- The library supports multiple AI providers and gateways: direct Anthropic/Google/OpenAI, or routed through Vercel AI Gateway, OpenRouter, OpenCode Zen, or Cloudflare AI Gateway.

- This is an important architectural boundary: the execution engine talks to an internal model abstraction, not directly to one provider everywhere.

5. Assertion subsystem

- src/assertion.ts implements the multi-model verification path.

- Assertions run primarily through a Claude + Gemini consensus flow, with an arbiter model resolving disagreements when configured to do so.

- This subsystem is separate from step execution: step automation gets the UI into a state, while assertions evaluate whether that state matches the expectation.

6. CUA mode is a distinct execution branch

- src/cua/ contains a separate computer-use-agent pipeline: actions.ts, client.ts, loop.ts, prompts.ts, and index.ts.

- In CUA mode, Passmark uses OpenAI’s Responses API with the built-in computer tool and visual/screenshot-driven actions instead of accessibility-snapshot-based automation.

- This path deliberately bypasses Redis step caching because coordinate-based actions are not portable across viewports.

7. Video assertions and artifact handling

- src/video.ts adds the “record the whole run and evaluate against video” capability.

- The runtime records Playwright screencast output, stores it under a configurable upload directory, and sends the resulting .webm to Gemini’s Files API for evaluation.

- This path is a special-case external integration: it always depends on Google access even if the rest of the run uses a different AI gateway.

8. Email and external test fixtures

- src/email.ts and src/providers/emailsink.ts implement a pluggable email provider boundary.

- Email extraction is resolved through the configured provider and used by placeholders like {{email.*}}, which lets tests interact with OTP/verification flows without hardcoding mailbox logic.

9. Prompting, extraction, and dynamic data

- src/prompts/, src/cua/prompts.ts, and src/extract.ts contain the prompt templates and AI extraction helpers used by execution, assertions, wait conditions, and structured data extraction.

- The placeholder system is another important data-flow boundary: {{run.*}} is per-test randomness, {{global.*}} is shared across an execution, {{data.*}} comes from Redis/project data, and {{email.*}} is lazily resolved from mail.

10. Observability and runtime diagnostics

- src/instrumentation.ts wraps AI calls with tracing spans, and src/logger.ts provides structured logging.

- Telemetry is optional and controlled through configuration or env vars, with Axiom/OpenTelemetry as the main observability path.

- This layer does not affect test semantics, but it is a key architectural boundary for debugging flaky AI behavior.

11. Security and controlled script execution

- src/utils/secure-script-runner.ts and src/utils/playwright-best-practices.ts show that generated or AI-assisted actions are constrained by an allowlisted execution surface.

- src/utils/tab-manager.ts manages browser tab/window context, which matters when AI-driven steps open new pages or switch targets.

- Together these utilities reduce the risk of letting AI output arbitrary code or unstable browser behavior.

12. Error, type, and test structure

- src/errors.ts, src/types.ts, and src/models.ts centralize the domain contracts and failure modes used across the library.

- The src/__tests__/ tree includes unit and integration coverage for config, caching, CUA flows, email, logging, secure script execution, file upload, and utility behavior.

- For an architecture diagram, this repo should be drawn as a Playwright-based testing engine with separate branches for step execution, CUA execution, assertions, caching/state, and observability, all behind one public configuration/API surface.