How ctrf-io/ai-test-reporter works
Project type and purpose
• This is a Node.js + TypeScript command-line tool (CLI) named “AI Test Reporter” (package name: ai-ctrf).
• Its purpose is to read a CTRF JSON test report, analyze every failing test via one of many AI model providers, optionally consolidate those AI summaries, and write back an augmented CTRF report.
File structure
• Top-level files: ESLint/Prettier configs, GitHub workflows, CONTRIBUTING, LICENSE, README.
• package.json / tsconfig.json define a Node CLI written in TS.
• /src holds the entire application source:
• index.ts – CLI entry point and command dispatcher
• common.ts, constants.ts – shared utilities/config
• consolidated-summary.ts – logic to merge multiple AI responses into a high-level overview
• models/ – one adapter per AI provider (azure-openai.ts, claude.ts, deepseek.ts, gemini.ts, grok.ts, mistral.ts, openai.ts, openrouter.ts, perplexity.ts)
README for additional insights
• The README confirms single-process CLI usage via npx ai-ctrf <provider> <path>.
• Each provider module wraps HTTP calls to an external AI API.
• It supports options like model, systemPrompt, temperature, maxTokens, consolidate, etc.
• Input: CTRF JSON test report; Output: the same CTRF JSON augmented with an “ai” field per failing test (and optionally a consolidated summary).
Main components
a. CLI Entry Point (index.ts)
b. Core Orchestrator
• File Reader / JSON Parser
• Test Filter (extract failed tests)
• Request Batcher (respects --maxMessages)
• Consolidated Summary Module
c. Provider Adapters (Strategy modules under src/models)
• One adapter per AI service: OpenAI, Azure, Claude, Gemini, Mistral, Grok, DeepSeek, Perplexity, OpenRouter
• Each adapter implements a common interface (prompt construction, HTTP request, parse response)
d. External AI Services
• REST/HTTP endpoints for each provider
e. File Writer
• Merge AI responses into original CTRF object
• Serialize & overwrite or output to new file
f. Configuration / Environment
• ENV vars (API keys, endpoints, deployment names)
• CLI flags
Relationships and interactions (data flow)
1. User invokes CLI (npx ai-ctrf <provider> report.json)
2. Index.ts parses CLI arguments → selects provider adapter → loads config
3. File Reader loads report.json → JSON parser produces Report object
4. Core Orchestrator filters failed tests → for each chunk (maxMessages) → calls Adapter.sendRequests()
5. Adapter builds prompt + sends HTTP request to External AI Service → receives AI summary(s)
6. Core Orchestrator attaches AI summaries to each test object
7. If --consolidate enabled: consolidated-summary.ts processes all summaries → produces high-level overview
8. File Writer serializes augmented Report → writes back to disk
9. CLI logs output (optional)
Architectural patterns and design principles
• Strategy Pattern: each AI provider implements a common interface (sendRequests, buildPrompt).
• Single responsibility: parsing, provider communication, response consolidation, file I/O are separate modules.
• Plugin-style extensibility: adding a new provider only requires a new module in src/models.
• Config-driven: environment variables + CLI flags drive behavior.
Key labels and elements
• “--maxMessages,” “--consolidate” flags influencing batching & consolidation
• Illustration of Strategy interface used by all provider modules
Final recommendation
• With key technology: Node.js, TypeScript, HTTP/REST.
• Keep it concise but complete—each internal module and external dependency should be represented.