How brentkastner/ai-qa-framework works
1. Overall shape
This repository is a Python 3.12+ autonomous QA framework centered on a four-stage pipeline: crawl → plan → execute → report. The system takes a target URL, inspects a live site in a real browser, uses AI to generate tests, runs those tests with Playwright, and produces persistent coverage and report artifacts.
2. Entry point and orchestration
The main operational boundary is the CLI in src/cli.py, which drives the pipeline through src/orchestrator.py. The CLI exposes both full-run and stage-by-stage commands, plus coverage and hint management. The orchestrator is the coordination layer that sequences subsystems and connects generated site data, test plans, execution results, and reporting output.
3. Browser and site discovery subsystem
Crawling lives under src/crawler/ and is responsible for inspecting the application as a user would. It appears to handle page discovery, element extraction, form analysis, and SPA-specific behavior. This is the source of the framework’s site model: it turns a live website into structured data that downstream planning can consume.
4. Site modeling and persisted discovery state
The discovered structure is represented in src/models/site_model.py and persisted under .qa-framework/site_model/model.json per the README. This model is the canonical boundary between “what the crawler found” and “what the planner can reason about.” It lets the framework reuse site knowledge across runs instead of treating each execution as a blank slate.
5. AI integration boundary
AI usage is isolated in src/ai/, especially src/ai/client.py and prompt templates under src/ai/prompts/. The prompts are split by intent: planning, summary, fallback, evaluation, and auth. The framework supports different AI providers, with Anthropic and Ollama called out in the README, and falls back to template-based behavior when no provider is configured.
6. Planning subsystem
src/planner/ converts the site model, hints, auth context, and coverage history into a structured TestPlan. A schema validator enforces plan shape before execution. This is the main intelligence boundary: the planner decides what should be tested, in what order, and with what categories, rather than hardcoding test scripts.
7. Coverage memory and gap analysis
The coverage layer in src/coverage/ tracks what has already been exercised and what remains weakly covered. It includes gap analysis, scoring, a registry, and a visual baseline registry. The persisted registry under .qa-framework/coverage/registry.json enables the framework to focus future runs on uncovered or stale areas, making the system stateful across executions.
8. Execution engine
src/executor/ runs the generated plan against the target site using Playwright-backed browser actions. The executor is split into action running, selector resolution, assertion checking, evidence collection, and fallback handling. This subsystem is where generated intent becomes real browser interaction, and it is also where selector breakage and dynamic UI recovery are handled.
9. Authentication and dynamic context
Authentication support is isolated in src/auth/smart_auth.py, with model/config support in src/models/config.py. The README shows credential injection via env vars and login URLs, so auth is treated as a reusable precondition for planning and execution rather than embedded into individual tests. This boundary matters because authenticated flows change both crawl visibility and test generation.
10. Reporting and regression detection
Results flow into src/reporter/, which emits both HTML and JSON reports and performs regression detection. The output goes to qa-reports/ and is meant to be both human-readable and machine-consumable. Reporting is also where AI-generated summaries and visual evidence are surfaced, making it the presentation boundary for the whole run.
11. Shared domain models
src/models/ holds the core data contracts: configuration, coverage, test plans, test results, visual baselines, and site model structures. These types are the glue between subsystems and are important for architecture diagrams because they define the stable handoff points between crawl, plan, execute, and report stages.
12. Supporting utilities and operational concerns
src/utils/browser_stealth.py suggests anti-detection or browser hardening when crawling/executing against real sites. src/url_utils.py likely provides normalization and URL handling for consistent site discovery. The repo also includes CI, tests, and example configs, indicating a strong emphasis on deterministic behavior around an inherently dynamic workflow.
13. End-to-end data flow
A typical run starts with a target URL and optional hints/auth config, crawls the site into a persisted model, feeds that model plus coverage state into the planner, executes the generated plan in a browser with evidence collection and recovery, and finally writes artifacts and reports. The key persistent stores are .qa-framework/ for internal state and qa-reports/ for run outputs.
14. Architectural takeaway
This is not a simple test runner; it is a stateful AI-assisted testing pipeline. The major boundaries are: browser discovery, AI planning, browser execution, and reporting, with coverage memory and auth as cross-cutting state. Any diagram should show the live website and browser on one side, the AI provider as a separate external service, and the persisted framework state as the bridge across runs.
Main entry points: executor.py Executor · orchestrator.py Orchestrator · crawler.py Crawler · planner.py Planner
Components 26
Entry & orchestration
CLI and top-level run coordination
- CLI · Command-line interface for full runs and staged operations
src/cli.py - Orchestrator · Sequences crawl, plan, execute, and report stages
src/orchestrator.py
Discovery
Browser crawling and site modeling
- Crawler · Browser-based site inspection and discovery
src/crawler/crawler.py - Elements · Extracts interactive elements from pages
src/crawler/element_extractor.py - Forms · Analyzes form structure and inputs
src/crawler/form_analyzer.py - SPA handler · Handles single-page app behavior during discovery
src/crawler/spa_handler.py
AI & planning
Provider integration, prompts, planning, and coverage memory
- AI client · Provider-facing client for AI-backed generation and summaries
src/ai/client.py - Prompts · Planning, summary, fallback, evaluation, and auth templates
src/ai/prompts/planning.py - Planner · Builds structured test plans from site, hints, auth, and coverage
src/planner/planner.py - Schema check · Validates generated plans before execution
src/planner/schema_validator.py - Coverage · Tracks exercised areas and uncovered gaps
src/coverage/registry.py - Gap analysis · Finds weakly covered or stale areas
src/coverage/gap_analyzer.py
Execution
Plan execution in a live browser
- Executor · Runs planned tests against the live site
src/executor/executor.py - Actions · Performs browser actions from the plan
src/executor/action_runner.py - Selector resolver · Resolves selectors and handles dynamic UI changes
src/executor/selector_resolver.py - Assertions · Checks expected outcomes during execution
src/executor/assertion_checker.py - Evidence · Collects screenshots, logs, and other run evidence
src/executor/evidence_collector.py - Fallback · Handles retries and degraded execution paths
src/executor/fallback.py
Reporting
Artifacts, regression checks, and final output
- Reporter · Coordinates final report generation and regression checks
src/reporter/reporter.py - HTML report · Human-readable report output
src/reporter/html_report.py - JSON report · Machine-readable report output
src/reporter/json_report.py - Regression detector · Detects regressions from current run results
src/reporter/regression_detector.py
Shared domain
Common data contracts and cross-cutting utilities
- Site model · Canonical discovered site structure persisted across runs
src/models/site_model.py - Config · Runtime configuration and auth-related settings
src/models/config.py - Test plan · Structured plan handed from planner to executor
src/models/test_plan.py - Test result · Execution outcome passed into reporting
src/models/test_result.py
Connections 28
- CLI → Orchestrator: starts · CLI invokes the top-level run coordinator
- Orchestrator → Crawler: crawl · Coordinates live browser discovery
- Crawler → Elements: extracts · Derives interactive elements from pages
- Crawler → Forms: analyzes · Builds form understanding for planning
- Crawler → SPA handler: handles · Adapts discovery for SPA behavior
- Crawler → Site model: persists · Stores discovered site structure
- Orchestrator → Planner: plan · Hands discovery state and context to planning
- Planner → AI client: uses · Requests AI-generated planning assistance
- AI client → Prompts: renders · Applies prompt templates for each AI task
- Planner → Schema check: validates · Checks plan shape before execution
- Planner → Coverage: reads · Uses prior coverage state to focus gaps
- Coverage → Gap analysis: computes · Derives uncovered or weakly covered areas
- Orchestrator → Executor: execute · Sends the approved plan to browser execution
- Executor → Actions: runs · Executes browser interactions
- Executor → Selector resolver: resolves · Finds and adapts selectors during runs
- Executor → Assertions: checks · Verifies expected behavior and outcomes
- Executor → Evidence: collects · Captures artifacts from execution
- Executor → Fallback: recovers · Handles failures and degraded paths
- Executor → Test result: emits · Produces structured execution results
- Orchestrator → Reporter: report · Passes results to reporting and regression checks
- Reporter → HTML report: writes · Generates human-readable output
- Reporter → JSON report: writes · Generates machine-readable output
- Reporter → Regression detector: analyzes · Checks for regressions across runs
- Orchestrator → Config: reads · Loads runtime configuration and auth settings
- Planner → Test plan: builds · Produces structured test plans
- Executor → Test plan: consumes · Executes the planned steps
- Reporter → Test result: consumes · Summarizes execution outcomes
- Orchestrator → Site model: reads · Reuses persisted site knowledge across runs