# brentkastner/ai-qa-framework architecture

> Architecture diagram of the GitHub repository brentkastner/ai-qa-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brentkastner/ai-qa-framework
- Repository: https://github.com/brentkastner/ai-qa-framework

## Overview

1. **Overall shape**
   This repository is a Python 3.12+ autonomous QA framework centered on a four-stage pipeline: **crawl → plan → execute → report**. The system takes a target URL, inspects a live site in a real browser, uses AI to generate tests, runs those tests with Playwright, and produces persistent coverage and report artifacts.

2. **Entry point and orchestration**
   The main operational boundary is the **CLI** in `src/cli.py`, which drives the pipeline through `src/orchestrator.py`. The CLI exposes both full-run and stage-by-stage commands, plus coverage and hint management. The orchestrator is the coordination layer that sequences subsystems and connects generated site data, test plans, execution results, and reporting output.

3. **Browser and site discovery subsystem**
   Crawling lives under `src/crawler/` and is responsible for inspecting the application as a user would. It appears to handle page discovery, element extraction, form analysis, and SPA-specific behavior. This is the source of the framework’s site model: it turns a live website into structured data that downstream planning can consume.

4. **Site modeling and persisted discovery state**
   The discovered structure is represented in `src/models/site_model.py` and persisted under `.qa-framework/site_model/model.json` per the README. This model is the canonical boundary between “what the crawler found” and “what the planner can reason about.” It lets the framework reuse site knowledge across runs instead of treating each execution as a blank slate.

5. **AI integration boundary**
   AI usage is isolated in `src/ai/`, especially `src/ai/client.py` and prompt templates under `src/ai/prompts/`. The prompts are split by intent: planning, summary, fallback, evaluation, and auth. The framework supports different AI providers, with Anthropic and Ollama called out in the README, and falls back to template-based behavior when no provider is configured.

6. **Planning subsystem**
   `src/planner/` converts the site model, hints, auth context, and coverage history into a structured `TestPlan`. A schema validator enforces plan shape before execution. This is the main intelligence boundary: the planner decides what should be tested, in what order, and with what categories, rather than hardcoding test scripts.

7. **Coverage memory and gap analysis**
   The coverage layer in `src/coverage/` tracks what has already been exercised and what remains weakly covered. It includes gap analysis, scoring, a registry, and a visual baseline registry. The persisted registry under `.qa-framework/coverage/registry.json` enables the framework to focus future runs on uncovered or stale areas, making the system stateful across executions.

8. **Execution engine**
   `src/executor/` runs the generated plan against the target site using Playwright-backed browser actions. The executor is split into action running, selector resolution, assertion checking, evidence collection, and fallback handling. This subsystem is where generated intent becomes real browser interaction, and it is also where selector breakage and dynamic UI recovery are handled.

9. **Authentication and dynamic context**
   Authentication support is isolated in `src/auth/smart_auth.py`, with model/config support in `src/models/config.py`. The README shows credential injection via env vars and login URLs, so auth is treated as a reusable precondition for planning and execution rather than embedded into individual tests. This boundary matters because authenticated flows change both crawl visibility and test generation.

10. **Reporting and regression detection**
   Results flow into `src/reporter/`, which emits both HTML and JSON reports and performs regression detection. The output goes to `qa-reports/` and is meant to be both human-readable and machine-consumable. Reporting is also where AI-generated summaries and visual evidence are surfaced, making it the presentation boundary for the whole run.

11. **Shared domain models**
   `src/models/` holds the core data contracts: configuration, coverage, test plans, test results, visual baselines, and site model structures. These types are the glue between subsystems and are important for architecture diagrams because they define the stable handoff points between crawl, plan, execute, and report stages.

12. **Supporting utilities and operational concerns**
   `src/utils/browser_stealth.py` suggests anti-detection or browser hardening when crawling/executing against real sites. `src/url_utils.py` likely provides normalization and URL handling for consistent site discovery. The repo also includes CI, tests, and example configs, indicating a strong emphasis on deterministic behavior around an inherently dynamic workflow.

13. **End-to-end data flow**
   A typical run starts with a target URL and optional hints/auth config, crawls the site into a persisted model, feeds that model plus coverage state into the planner, executes the generated plan in a browser with evidence collection and recovery, and finally writes artifacts and reports. The key persistent stores are `.qa-framework/` for internal state and `qa-reports/` for run outputs.

14. **Architectural takeaway**
   This is not a simple test runner; it is a **stateful AI-assisted testing pipeline**. The major boundaries are: browser discovery, AI planning, browser execution, and reporting, with coverage memory and auth as cross-cutting state. Any diagram should show the live website and browser on one side, the AI provider as a separate external service, and the persisted framework state as the bridge across runs.

Main entry points: [`src/executor/executor.py`](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/executor.py) (Executor), [`src/orchestrator.py`](https://github.com/brentkastner/ai-qa-framework/blob/main/src/orchestrator.py) (Orchestrator), [`src/crawler/crawler.py`](https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/crawler.py) (Crawler), [`src/planner/planner.py`](https://github.com/brentkastner/ai-qa-framework/blob/main/src/planner/planner.py) (Planner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_entry["Entry & orchestration"]
  node_cli["CLI<br/>entrypoint<br/>[cli.py]"]
  node_orchestrator["Orchestrator<br/>coordination<br/>[orchestrator.py]"]
end

subgraph group_grp_discovery["Discovery"]
  node_crawler["Crawler<br/>discovery<br/>[crawler.py]"]
  node_elements["Elements<br/>extraction"]
  node_forms["Forms<br/>analysis<br/>[form_analyzer.py]"]
  node_spa["SPA handler<br/>navigation<br/>[spa_handler.py]"]
end

subgraph group_grp_intelligence["AI & planning"]
  node_ai_client["AI client<br/>provider adapter<br/>[client.py]"]
  node_prompts["Prompts<br/>prompt bundle<br/>[planning.py]"]
  node_planner["Planner<br/>test planning<br/>[planner.py]"]
  node_schema_validator["Schema check<br/>validation"]
  node_coverage[("Coverage<br/>stateful analysis<br/>[registry.py]")]
  node_gap_analyzer["Gap analysis<br/>[gap_analyzer.py]"]
end

subgraph group_grp_execution["Execution"]
  node_executor["Executor<br/>browser execution<br/>[executor.py]"]
  node_actions["Actions<br/>interaction runner<br/>[action_runner.py]"]
  node_selectors["Selector resolver<br/>resilience"]
  node_assertions["Assertions<br/>verification"]
  node_evidence["Evidence<br/>artifact collection"]
  node_fallback["Fallback<br/>recovery<br/>[fallback.py]"]
end

subgraph group_grp_output["Reporting"]
  node_reporter["Reporter<br/>output<br/>[reporter.py]"]
  node_html_report["HTML report<br/>presentation<br/>[html_report.py]"]
  node_json_report["JSON report<br/>artifact<br/>[json_report.py]"]
  node_regression["Regression detector<br/>analysis"]
end

subgraph group_grp_shared["Shared domain"]
  node_site_model[("Site model<br/>domain model<br/>[site_model.py]")]
  node_config["Config<br/>[config.py]"]
  node_test_plan["Test plan<br/>domain model<br/>[test_plan.py]"]
  node_test_result["Test result<br/>domain model<br/>[test_result.py]"]
end

node_cli -->|"starts"| node_orchestrator
node_orchestrator -->|"crawl"| node_crawler
node_crawler -->|"extracts"| node_elements
node_crawler -->|"analyzes"| node_forms
node_crawler -->|"handles"| node_spa
node_crawler -->|"persists"| node_site_model
node_orchestrator -->|"plan"| node_planner
node_planner -->|"uses"| node_ai_client
node_ai_client -->|"renders"| node_prompts
node_planner -->|"validates"| node_schema_validator
node_planner -->|"reads"| node_coverage
node_coverage -->|"computes"| node_gap_analyzer
node_orchestrator -->|"execute"| node_executor
node_executor -->|"runs"| node_actions
node_executor -->|"resolves"| node_selectors
node_executor -->|"checks"| node_assertions
node_executor -->|"collects"| node_evidence
node_executor -->|"recovers"| node_fallback
node_executor -->|"emits"| node_test_result
node_orchestrator -->|"report"| node_reporter
node_reporter -->|"writes"| node_html_report
node_reporter -->|"writes"| node_json_report
node_reporter -->|"analyzes"| node_regression
node_orchestrator -->|"reads"| node_config
node_planner -->|"builds"| node_test_plan
node_executor -->|"consumes"| node_test_plan
node_reporter -->|"consumes"| node_test_result
node_orchestrator -->|"reads"| node_site_model

click node_cli "https://github.com/brentkastner/ai-qa-framework/blob/main/src/cli.py"
click node_orchestrator "https://github.com/brentkastner/ai-qa-framework/blob/main/src/orchestrator.py"
click node_crawler "https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/crawler.py"
click node_elements "https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/element_extractor.py"
click node_forms "https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/form_analyzer.py"
click node_spa "https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/spa_handler.py"
click node_site_model "https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/site_model.py"
click node_ai_client "https://github.com/brentkastner/ai-qa-framework/blob/main/src/ai/client.py"
click node_prompts "https://github.com/brentkastner/ai-qa-framework/blob/main/src/ai/prompts/planning.py"
click node_planner "https://github.com/brentkastner/ai-qa-framework/blob/main/src/planner/planner.py"
click node_schema_validator "https://github.com/brentkastner/ai-qa-framework/blob/main/src/planner/schema_validator.py"
click node_coverage "https://github.com/brentkastner/ai-qa-framework/blob/main/src/coverage/registry.py"
click node_gap_analyzer "https://github.com/brentkastner/ai-qa-framework/blob/main/src/coverage/gap_analyzer.py"
click node_executor "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/executor.py"
click node_actions "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/action_runner.py"
click node_selectors "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/selector_resolver.py"
click node_assertions "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/assertion_checker.py"
click node_evidence "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/evidence_collector.py"
click node_fallback "https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/fallback.py"
click node_reporter "https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/reporter.py"
click node_html_report "https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/html_report.py"
click node_json_report "https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/json_report.py"
click node_regression "https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/regression_detector.py"
click node_config "https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/config.py"
click node_test_plan "https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/test_plan.py"
click node_test_result "https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/test_result.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_orchestrator toneBlue
class node_crawler,node_elements,node_forms,node_spa toneAmber
class node_ai_client,node_prompts,node_planner,node_schema_validator,node_coverage,node_gap_analyzer toneMint
class node_executor,node_actions,node_selectors,node_assertions,node_evidence,node_fallback toneRose
class node_reporter,node_html_report,node_json_report,node_regression toneIndigo
class node_site_model,node_config,node_test_plan,node_test_result toneTeal
```

## Components

### Entry & orchestration

CLI and top-level run coordination

- [CLI](https://github.com/brentkastner/ai-qa-framework/blob/main/src/cli.py): entrypoint · Command-line interface for full runs and staged operations · `src/cli.py`
- [Orchestrator](https://github.com/brentkastner/ai-qa-framework/blob/main/src/orchestrator.py): coordination · Sequences crawl, plan, execute, and report stages · `src/orchestrator.py`

### Discovery

Browser crawling and site modeling

- [Crawler](https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/crawler.py): discovery · Browser-based site inspection and discovery · `src/crawler/crawler.py`
- [Elements](https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/element_extractor.py): extraction · Extracts interactive elements from pages · `src/crawler/element_extractor.py`
- [Forms](https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/form_analyzer.py): analysis · Analyzes form structure and inputs · `src/crawler/form_analyzer.py`
- [SPA handler](https://github.com/brentkastner/ai-qa-framework/blob/main/src/crawler/spa_handler.py): navigation · Handles single-page app behavior during discovery · `src/crawler/spa_handler.py`

### AI & planning

Provider integration, prompts, planning, and coverage memory

- [AI client](https://github.com/brentkastner/ai-qa-framework/blob/main/src/ai/client.py): provider adapter · Provider-facing client for AI-backed generation and summaries · `src/ai/client.py`
- [Prompts](https://github.com/brentkastner/ai-qa-framework/blob/main/src/ai/prompts/planning.py): prompt bundle · Planning, summary, fallback, evaluation, and auth templates · `src/ai/prompts/planning.py`
- [Planner](https://github.com/brentkastner/ai-qa-framework/blob/main/src/planner/planner.py): test planning · Builds structured test plans from site, hints, auth, and coverage · `src/planner/planner.py`
- [Schema check](https://github.com/brentkastner/ai-qa-framework/blob/main/src/planner/schema_validator.py): validation · Validates generated plans before execution · `src/planner/schema_validator.py`
- [Coverage](https://github.com/brentkastner/ai-qa-framework/blob/main/src/coverage/registry.py): stateful analysis · Tracks exercised areas and uncovered gaps · `src/coverage/registry.py`
- [Gap analysis](https://github.com/brentkastner/ai-qa-framework/blob/main/src/coverage/gap_analyzer.py): analysis · Finds weakly covered or stale areas · `src/coverage/gap_analyzer.py`

### Execution

Plan execution in a live browser

- [Executor](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/executor.py): browser execution · Runs planned tests against the live site · `src/executor/executor.py`
- [Actions](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/action_runner.py): interaction runner · Performs browser actions from the plan · `src/executor/action_runner.py`
- [Selector resolver](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/selector_resolver.py): resilience · Resolves selectors and handles dynamic UI changes · `src/executor/selector_resolver.py`
- [Assertions](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/assertion_checker.py): verification · Checks expected outcomes during execution · `src/executor/assertion_checker.py`
- [Evidence](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/evidence_collector.py): artifact collection · Collects screenshots, logs, and other run evidence · `src/executor/evidence_collector.py`
- [Fallback](https://github.com/brentkastner/ai-qa-framework/blob/main/src/executor/fallback.py): recovery · Handles retries and degraded execution paths · `src/executor/fallback.py`

### Reporting

Artifacts, regression checks, and final output

- [Reporter](https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/reporter.py): output · Coordinates final report generation and regression checks · `src/reporter/reporter.py`
- [HTML report](https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/html_report.py): presentation · Human-readable report output · `src/reporter/html_report.py`
- [JSON report](https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/json_report.py): artifact · Machine-readable report output · `src/reporter/json_report.py`
- [Regression detector](https://github.com/brentkastner/ai-qa-framework/blob/main/src/reporter/regression_detector.py): analysis · Detects regressions from current run results · `src/reporter/regression_detector.py`

### Shared domain

Common data contracts and cross-cutting utilities

- [Site model](https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/site_model.py): domain model · Canonical discovered site structure persisted across runs · `src/models/site_model.py`
- [Config](https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/config.py): shared config · Runtime configuration and auth-related settings · `src/models/config.py`
- [Test plan](https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/test_plan.py): domain model · Structured plan handed from planner to executor · `src/models/test_plan.py`
- [Test result](https://github.com/brentkastner/ai-qa-framework/blob/main/src/models/test_result.py): domain model · Execution outcome passed into reporting · `src/models/test_result.py`

## Connections

- CLI → Orchestrator: starts · CLI invokes the top-level run coordinator
- Orchestrator → Crawler: crawl · Coordinates live browser discovery
- Crawler → Elements: extracts · Derives interactive elements from pages
- Crawler → Forms: analyzes · Builds form understanding for planning
- Crawler → SPA handler: handles · Adapts discovery for SPA behavior
- Crawler → Site model: persists · Stores discovered site structure
- Orchestrator → Planner: plan · Hands discovery state and context to planning
- Planner → AI client: uses · Requests AI-generated planning assistance
- AI client → Prompts: renders · Applies prompt templates for each AI task
- Planner → Schema check: validates · Checks plan shape before execution
- Planner → Coverage: reads · Uses prior coverage state to focus gaps
- Coverage → Gap analysis: computes · Derives uncovered or weakly covered areas
- Orchestrator → Executor: execute · Sends the approved plan to browser execution
- Executor → Actions: runs · Executes browser interactions
- Executor → Selector resolver: resolves · Finds and adapts selectors during runs
- Executor → Assertions: checks · Verifies expected behavior and outcomes
- Executor → Evidence: collects · Captures artifacts from execution
- Executor → Fallback: recovers · Handles failures and degraded paths
- Executor → Test result: emits · Produces structured execution results
- Orchestrator → Reporter: report · Passes results to reporting and regression checks
- Reporter → HTML report: writes · Generates human-readable output
- Reporter → JSON report: writes · Generates machine-readable output
- Reporter → Regression detector: analyzes · Checks for regressions across runs
- Orchestrator → Config: reads · Loads runtime configuration and auth settings
- Planner → Test plan: builds · Produces structured test plans
- Executor → Test plan: consumes · Executes the planned steps
- Reporter → Test result: consumes · Summarizes execution outcomes
- Orchestrator → Site model: reads · Reuses persisted site knowledge across runs
