# bug0inc/passmark architecture

> Architecture diagram of the GitHub repository bug0inc/passmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bug0inc/passmark
- Repository: https://github.com/bug0inc/passmark

## Overview

1. **Library façade and public API**
   - `src/index.ts` is the main entrypoint and re-exports the package surface: `configure()`, `runSteps()`, `runUserFlow()`, `assert()`, extraction helpers, email providers, and utility types.
   - This repo is not an app; it is a TypeScript Playwright testing library that sits inside a user’s Playwright test suite and drives a `page` object.

2. **Primary execution path: natural-language steps over Playwright**
   - The core subsystem is `runSteps()` in the `src/` execution modules, which converts step descriptions into browser actions against Playwright.
   - The data flow is: step request → cache lookup → AI-guided step execution or cached action replay → optional healing/fallback → result returned to the test.
   - `runUserFlow()` is the higher-level variant that treats an entire flow more like a single agentic task rather than a step-by-step script.

3. **Cache and state boundary**
   - `src/redis.ts` and `src/data-cache.ts` back Redis-based caching and cross-run data storage.
   - Successful single-step actions are cached by flow/step identity and replayed on later runs to reduce latency and cost.
   - The same Redis-backed layer also supports `{{global.*}}` and `{{data.*}}` placeholder resolution; without Redis, those features degrade or disable cleanly.

4. **AI model orchestration**
   - `src/config.ts`, `src/models.ts`, `src/constants.ts`, and `src/tools.ts` define the model slots, gateway selection, and the AI tool surface.
   - The library supports multiple AI providers and gateways: direct Anthropic/Google/OpenAI, or routed through Vercel AI Gateway, OpenRouter, OpenCode Zen, or Cloudflare AI Gateway.
   - This is an important architectural boundary: the execution engine talks to an internal model abstraction, not directly to one provider everywhere.

5. **Assertion subsystem**
   - `src/assertion.ts` implements the multi-model verification path.
   - Assertions run primarily through a Claude + Gemini consensus flow, with an arbiter model resolving disagreements when configured to do so.
   - This subsystem is separate from step execution: step automation gets the UI into a state, while assertions evaluate whether that state matches the expectation.

6. **CUA mode is a distinct execution branch**
   - `src/cua/` contains a separate computer-use-agent pipeline: `actions.ts`, `client.ts`, `loop.ts`, `prompts.ts`, and `index.ts`.
   - In CUA mode, Passmark uses OpenAI’s Responses API with the built-in `computer` tool and visual/screenshot-driven actions instead of accessibility-snapshot-based automation.
   - This path deliberately bypasses Redis step caching because coordinate-based actions are not portable across viewports.

7. **Video assertions and artifact handling**
   - `src/video.ts` adds the “record the whole run and evaluate against video” capability.
   - The runtime records Playwright screencast output, stores it under a configurable upload directory, and sends the resulting `.webm` to Gemini’s Files API for evaluation.
   - This path is a special-case external integration: it always depends on Google access even if the rest of the run uses a different AI gateway.

8. **Email and external test fixtures**
   - `src/email.ts` and `src/providers/emailsink.ts` implement a pluggable email provider boundary.
   - Email extraction is resolved through the configured provider and used by placeholders like `{{email.*}}`, which lets tests interact with OTP/verification flows without hardcoding mailbox logic.

9. **Prompting, extraction, and dynamic data**
   - `src/prompts/`, `src/cua/prompts.ts`, and `src/extract.ts` contain the prompt templates and AI extraction helpers used by execution, assertions, wait conditions, and structured data extraction.
   - The placeholder system is another important data-flow boundary: `{{run.*}}` is per-test randomness, `{{global.*}}` is shared across an execution, `{{data.*}}` comes from Redis/project data, and `{{email.*}}` is lazily resolved from mail.

10. **Observability and runtime diagnostics**
   - `src/instrumentation.ts` wraps AI calls with tracing spans, and `src/logger.ts` provides structured logging.
   - Telemetry is optional and controlled through configuration or env vars, with Axiom/OpenTelemetry as the main observability path.
   - This layer does not affect test semantics, but it is a key architectural boundary for debugging flaky AI behavior.

11. **Security and controlled script execution**
   - `src/utils/secure-script-runner.ts` and `src/utils/playwright-best-practices.ts` show that generated or AI-assisted actions are constrained by an allowlisted execution surface.
   - `src/utils/tab-manager.ts` manages browser tab/window context, which matters when AI-driven steps open new pages or switch targets.
   - Together these utilities reduce the risk of letting AI output arbitrary code or unstable browser behavior.

12. **Error, type, and test structure**
   - `src/errors.ts`, `src/types.ts`, and `src/models.ts` centralize the domain contracts and failure modes used across the library.
   - The `src/__tests__/` tree includes unit and integration coverage for config, caching, CUA flows, email, logging, secure script execution, file upload, and utility behavior.
   - For an architecture diagram, this repo should be drawn as a Playwright-based testing engine with separate branches for step execution, CUA execution, assertions, caching/state, and observability, all behind one public configuration/API surface.

Main entry points: [`src/models.ts`](https://github.com/bug0inc/passmark/blob/main/src/models.ts) (Model Abstraction), [`src/index.ts`](https://github.com/bug0inc/passmark/blob/main/src/index.ts) (API Facade), [`src/prompts/index.ts`](https://github.com/bug0inc/passmark/blob/main/src/prompts/index.ts) (Prompts), [`src/cua/index.ts`](https://github.com/bug0inc/passmark/blob/main/src/cua/index.ts) (CUA Mode)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public_api["Public API"]
  node_index["API Facade<br/>entrypoint<br/>[index.ts]"]
  node_config["Config<br/>[config.ts]"]
  node_types["Domain Types<br/>contracts<br/>[types.ts]"]
end

subgraph group_execution["Execution"]
  node_steps["Run Steps<br/>step engine"]
  node_user_flow["Run Flow<br/>flow engine"]
  node_assertion["Assertions<br/>verification<br/>[assertion.ts]"]
end

subgraph group_state["State & Data"]
  node_cache[("Run Cache<br/>redis cache<br/>[data-cache.ts]")]
  node_redis["Redis Layer<br/>redis access<br/>[redis.ts]"]
  node_email["Email Provider<br/>fixture provider<br/>[email.ts]"]
end

subgraph group_ai["AI Layer"]
  node_extract["Extraction<br/>structured extract<br/>[extract.ts]"]
  node_prompts["Prompts<br/>prompt templates<br/>[index.ts]"]
  node_tools["Tools<br/>tool surface<br/>[tools.ts]"]
  node_models["Model Abstraction<br/>model routing<br/>[models.ts]"]
end

subgraph group_special["Special Paths"]
  node_cua["CUA Mode<br/>agent branch<br/>[index.ts]"]
  node_video["Video Check<br/>artifact eval<br/>[video.ts]"]
end

subgraph group_ops["Ops & Safety"]
  node_instrumentation["Tracing<br/>telemetry<br/>[instrumentation.ts]"]
  node_logger["Logger<br/>logging<br/>[logger.ts]"]
  node_secure["Safe Runner<br/>safety util"]
  node_tabs["Tab Manager<br/>browser context<br/>[tab-manager.ts]"]
end

node_index -->|"configure"| node_config
node_index -->|"runSteps"| node_steps
node_index -->|"runUserFlow"| node_user_flow
node_index -->|"assert"| node_assertion
node_index -->|"extract helpers"| node_extract
node_steps -->|"lookup/replay"| node_cache
node_steps -->|"plan actions"| node_models
node_steps -->|"prompting"| node_prompts
node_steps -->|"tool use"| node_tools
node_steps -->|"guarded execution"| node_secure
node_steps -->|"browser context"| node_tabs
node_user_flow -->|"agent loop"| node_models
node_user_flow -->|"prompting"| node_prompts
node_assertion -->|"verify"| node_models
node_assertion -->|"verification prompts"| node_prompts
node_extract -->|"extract"| node_models
node_extract -->|"extraction prompts"| node_prompts
node_models -->|"provider selection"| node_config
node_cache -->|"store/load"| node_redis
node_cache -->|"placeholder data"| node_email
node_email -.->|"optional backing"| node_redis
node_cua -->|"Responses API"| node_models
node_cua -->|"visual prompts"| node_prompts
node_cua -->|"diagnostics"| node_logger
node_video -->|"Gemini eval"| node_models
node_video -->|"artifact logs"| node_logger
node_instrumentation -->|"trace AI"| node_models
node_logger -.->|"runtime logs"| node_index
node_secure -->|"context safety"| node_tabs

click node_index "https://github.com/bug0inc/passmark/blob/main/src/index.ts"
click node_config "https://github.com/bug0inc/passmark/blob/main/src/config.ts"
click node_types "https://github.com/bug0inc/passmark/blob/main/src/types.ts"
click node_assertion "https://github.com/bug0inc/passmark/blob/main/src/assertion.ts"
click node_extract "https://github.com/bug0inc/passmark/blob/main/src/extract.ts"
click node_prompts "https://github.com/bug0inc/passmark/blob/main/src/prompts/index.ts"
click node_tools "https://github.com/bug0inc/passmark/blob/main/src/tools.ts"
click node_models "https://github.com/bug0inc/passmark/blob/main/src/models.ts"
click node_cua "https://github.com/bug0inc/passmark/blob/main/src/cua/index.ts"
click node_video "https://github.com/bug0inc/passmark/blob/main/src/video.ts"
click node_cache "https://github.com/bug0inc/passmark/blob/main/src/data-cache.ts"
click node_redis "https://github.com/bug0inc/passmark/blob/main/src/redis.ts"
click node_email "https://github.com/bug0inc/passmark/blob/main/src/email.ts"
click node_instrumentation "https://github.com/bug0inc/passmark/blob/main/src/instrumentation.ts"
click node_logger "https://github.com/bug0inc/passmark/blob/main/src/logger.ts"
click node_secure "https://github.com/bug0inc/passmark/blob/main/src/utils/secure-script-runner.ts"
click node_tabs "https://github.com/bug0inc/passmark/blob/main/src/utils/tab-manager.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_index,node_config,node_types toneBlue
class node_steps,node_user_flow,node_assertion toneAmber
class node_cache,node_redis,node_email toneMint
class node_extract,node_prompts,node_tools,node_models toneRose
class node_cua,node_video toneIndigo
class node_instrumentation,node_logger,node_secure,node_tabs toneTeal
```

## Components

### Public API

Package entrypoints and the main surface consumed by user Playwright test suites.

- [API Facade](https://github.com/bug0inc/passmark/blob/main/src/index.ts): entrypoint · Main package export surface for configuration, execution, assertions, extraction, providers, and utilities. · `src/index.ts`
- [Config](https://github.com/bug0inc/passmark/blob/main/src/config.ts): runtime config · Configuration and gateway selection for models, telemetry, cache, and providers. · `src/config.ts`
- [Domain Types](https://github.com/bug0inc/passmark/blob/main/src/types.ts): contracts · Shared library contracts and utility types used across execution and integrations. · `src/types.ts`

### Execution

Primary step and flow execution paths over a Playwright page.

- **Run Steps**: step engine · Core step execution pipeline that turns natural-language steps into browser actions.
- **Run Flow**: flow engine · Higher-level agentic flow execution built on the same Playwright-driven runtime.
- [Assertions](https://github.com/bug0inc/passmark/blob/main/src/assertion.ts): verification · Multi-model verification path for checking UI state against expectations. · `src/assertion.ts`

### State & Data

Redis-backed caching, placeholder resolution, and external fixture providers.

- [Run Cache](https://github.com/bug0inc/passmark/blob/main/src/data-cache.ts): redis cache · Redis-backed caching for successful step replay and shared run data. · `src/data-cache.ts`
- [Redis Layer](https://github.com/bug0inc/passmark/blob/main/src/redis.ts): redis access · Redis connection and storage boundary used by cache and placeholder resolution. · `src/redis.ts`
- [Email Provider](https://github.com/bug0inc/passmark/blob/main/src/email.ts): fixture provider · Pluggable email boundary for OTP and inbox-driven test flows. · `src/email.ts`

### AI Layer

Model selection, prompts, tools, extraction, and assertion orchestration.

- [Extraction](https://github.com/bug0inc/passmark/blob/main/src/extract.ts): structured extract · AI-assisted extraction helpers for structured data and wait-condition style reads. · `src/extract.ts`
- [Prompts](https://github.com/bug0inc/passmark/blob/main/src/prompts/index.ts): prompt templates · Shared prompt templates used by execution, extraction, and verification flows. · `src/prompts/index.ts`
- [Tools](https://github.com/bug0inc/passmark/blob/main/src/tools.ts): tool surface · AI tool definitions exposed to model calls for browser-oriented actions. · `src/tools.ts`
- [Model Abstraction](https://github.com/bug0inc/passmark/blob/main/src/models.ts): model routing · Provider and gateway abstraction for direct or routed model access. · `src/models.ts`

### Special Paths

Alternative execution and artifact-driven evaluation branches.

- [CUA Mode](https://github.com/bug0inc/passmark/blob/main/src/cua/index.ts): agent branch · Separate computer-use-agent pipeline using OpenAI Responses and visual actions. · `src/cua/index.ts`
- [Video Check](https://github.com/bug0inc/passmark/blob/main/src/video.ts): artifact eval · Screencast recording and Gemini file-based evaluation of a full run video. · `src/video.ts`

### Ops & Safety

Logging, tracing, browser-context safety, and controlled execution helpers.

- [Tracing](https://github.com/bug0inc/passmark/blob/main/src/instrumentation.ts): telemetry · Tracing spans around AI calls and key runtime operations. · `src/instrumentation.ts`
- [Logger](https://github.com/bug0inc/passmark/blob/main/src/logger.ts): logging · Structured logging for runtime diagnostics and debugging flaky behavior. · `src/logger.ts`
- [Safe Runner](https://github.com/bug0inc/passmark/blob/main/src/utils/secure-script-runner.ts): safety util · Constrained script execution surface for AI-assisted or generated actions. · `src/utils/secure-script-runner.ts`
- [Tab Manager](https://github.com/bug0inc/passmark/blob/main/src/utils/tab-manager.ts): browser context · Tab and window context management for multi-page browser interactions. · `src/utils/tab-manager.ts`

## Connections

- API Facade → Config: configure · Public configuration is created and consumed through the package facade.
- API Facade → Run Steps: runSteps · Primary step execution API exported by the package.
- API Facade → Run Flow: runUserFlow · Higher-level flow execution API exported by the package.
- API Facade → Assertions: assert · Assertion API exported to user test suites.
- API Facade → Extraction: extract helpers · Public extraction helpers are re-exported from the facade.
- Run Steps → Run Cache: lookup/replay · Step execution checks cache for prior successful actions before re-running AI.
- Run Steps → Model Abstraction: plan actions · Execution delegates decision-making to the internal model abstraction.
- Run Steps → Prompts: prompting · Step planning and healing use shared prompt templates.
- Run Steps → Tools: tool use · Model-guided execution relies on the browser tool surface.
- Run Steps → Safe Runner: guarded execution · Generated actions are constrained by safe execution helpers.
- Run Steps → Tab Manager: browser context · Actions may switch or manage tabs and windows.
- Run Flow → Model Abstraction: agent loop · Flow execution uses the same model abstraction for agentic decisions.
- Run Flow → Prompts: prompting · Flow-level prompts steer the higher-level agent behavior.
- Assertions → Model Abstraction: verify · Assertions run through multiple model calls and consensus/arbiter logic.
- Assertions → Prompts: verification prompts · Assertion reasoning is driven by dedicated prompt templates.
- Extraction → Model Abstraction: extract · Structured extraction uses the model abstraction for parsing and normalization.
- Extraction → Prompts: extraction prompts · Extraction helpers rely on prompt templates for schema-aware output.
- Model Abstraction → Config: provider selection · Model routing and gateway choice are controlled by configuration.
- Run Cache → Redis Layer: store/load · Cache and shared data are backed by Redis storage.
- Run Cache → Email Provider: placeholder data · Cached runtime state supports email-backed placeholder resolution.
- Email Provider → Redis Layer: optional backing · Email-related runtime state may participate in Redis-backed resolution.
- CUA Mode → Model Abstraction: Responses API · CUA mode uses a separate model/client path built around OpenAI Responses.
- CUA Mode → Prompts: visual prompts · CUA loop and action generation use dedicated prompt templates.
- CUA Mode → Logger: diagnostics · CUA execution emits logs for agent behavior and debugging.
- Video Check → Model Abstraction: Gemini eval · Video-based evaluation sends recorded artifacts to Gemini files/evaluation.
- Video Check → Logger: artifact logs · Video recording and upload flow emits runtime diagnostics.
- Tracing → Model Abstraction: trace AI · Tracing wraps model calls and other important runtime spans.
- Logger → API Facade: runtime logs · Logging supports the public runtime surface and its consumers.
- Safe Runner → Tab Manager: context safety · Safe execution helpers work alongside browser tab/window management.