# sunny-arch-82/ai-evaluation-and-regression-testing-framework architecture

> Architecture diagram of the GitHub repository sunny-arch-82/ai-evaluation-and-regression-testing-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework
- Repository: https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework

## Overview

EvalForge is a local evaluation and regression framework, not a chatbot: users provide labeled cases and a YAML run configuration, then select a target adapter. The runner executes cases, validates target outputs, scores them, and saves immutable run records and reports. Users can then compare compatible runs and apply fail-closed gates. The graph also preserves citation, retrieval, structured-output, tool-trace, cost, latency, and offline/HTTP/Python adapter capabilities. Relationships are based on the README and shown source; unsampled subsystems are included only where documented.

Main entry points: [`src/evalforge/runner.py`](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/runner.py) (Sequential runner), [`src/evalforge/cli.py`](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/cli.py) (CLI commands), [`src/evalforge/targets.py`](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py) (Target adapters), [`src/evalforge/comparison.py`](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/comparison.py) (Baseline comparison)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_contracts_inputs["Contracts and inputs"]
  node_config["Run configuration"]
  node_cases["Golden cases"]
  node_contracts["Case and output contracts<br/>[schemas.py]"]
  node_saved_outputs["Saved responses"]
end

subgraph group_target_adapters["Target adapters"]
  node_target_dispatch["Target adapters<br/>[targets.py]"]
  node_demo["Demo behaviors<br/>[demo.py]"]
  node_python_target["Python target<br/>[targets.py]"]
  node_offline_target["Saved-output target<br/>[targets.py]"]
  node_http_target["HTTP target<br/>[targets.py]"]
end

subgraph group_run_scoring["Run and scoring"]
  node_cli["CLI commands<br/>[cli.py]"]
  node_runner["Sequential runner<br/>[runner.py]"]
  node_metrics["Metrics and aggregation<br/>[metrics.py]"]
end

subgraph group_comparison_gates["Comparison and gates"]
  node_comparison["Baseline comparison<br/>[comparison.py]"]
  node_gates["Regression gates<br/>[comparison.py]"]
end

subgraph group_records_reports["Records and reports"]
  node_io["Config and run IO<br/>[io.py]"]
  node_run_records["Run records<br/>[runner.py]"]
  node_reports["Run reports<br/>[reporting.py]"]
  node_comparison_reports["Comparison reports<br/>[reporting.py]"]
end

node_user(("Evaluator"))
node_python_system["User Python system"]
node_http_service["HTTP evaluation endpoint"]

node_user -->|"invokes"| node_cli
node_config -->|"configures"| node_cli
node_cases -->|"loaded as"| node_io
node_cli -->|"starts run"| node_runner
node_runner -->|"loads and saves"| node_io
node_io -->|"validates"| node_contracts
node_runner -->|"validates output"| node_contracts
node_runner -->|"builds and calls"| node_target_dispatch
node_target_dispatch -->|"calls demo"| node_demo
node_target_dispatch -->|"dispatches Python"| node_python_target
node_target_dispatch -->|"dispatches offline"| node_offline_target
node_target_dispatch -->|"dispatches HTTP"| node_http_target
node_python_target -->|"invokes"| node_python_system
node_http_target -->|"posts cases"| node_http_service
node_offline_target -->|"reads"| node_saved_outputs
node_runner -->|"scores and aggregates"| node_metrics
node_runner -->|"writes"| node_run_records
node_runner -->|"generates"| node_reports
node_cli -->|"regenerates"| node_reports
node_cli -->|"starts comparison"| node_comparison
node_comparison -->|"loads runs"| node_io
node_comparison -->|"validates gate config"| node_contracts
node_comparison -->|"evaluates"| node_gates
node_cli -->|"writes comparison"| node_comparison_reports

click node_config "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/tree/main/configs"
click node_cases "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/tree/main/data/sample"
click node_contracts "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/schemas.py"
click node_cli "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/cli.py"
click node_io "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/io.py"
click node_runner "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/runner.py"
click node_target_dispatch "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py"
click node_demo "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/demo.py"
click node_python_target "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py"
click node_offline_target "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py"
click node_http_target "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py"
click node_saved_outputs "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/data/sample/saved_outputs.jsonl"
click node_metrics "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/metrics.py"
click node_run_records "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/runner.py"
click node_reports "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/reporting.py"
click node_comparison "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/comparison.py"
click node_gates "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/comparison.py"
click node_comparison_reports "https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/reporting.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_cases,node_contracts,node_saved_outputs,node_python_system toneBlue
class node_target_dispatch,node_demo,node_python_target,node_offline_target,node_http_target toneAmber
class node_cli,node_runner,node_metrics toneMint
class node_comparison,node_gates toneRose
class node_io,node_run_records,node_reports,node_comparison_reports,node_user,node_http_service toneIndigo
```

## Components

### Contracts and inputs

- [Run configuration](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/tree/main/configs): `configs`
- [Golden cases](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/tree/main/data/sample): `data/sample`
- [Case and output contracts](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/schemas.py): `src/evalforge/schemas.py`
- [Saved responses](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/data/sample/saved_outputs.jsonl): `data/sample/saved_outputs.jsonl`

### Target adapters

- [Target adapters](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py): `src/evalforge/targets.py`
- [Demo behaviors](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/demo.py): `src/evalforge/demo.py`
- [Python target](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py): `src/evalforge/targets.py`
- [Saved-output target](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py): `src/evalforge/targets.py`
- [HTTP target](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/targets.py): `src/evalforge/targets.py`

### Run and scoring

- [CLI commands](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/cli.py): `src/evalforge/cli.py`
- [Sequential runner](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/runner.py): `src/evalforge/runner.py`
- [Metrics and aggregation](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/metrics.py): `src/evalforge/metrics.py`

### Comparison and gates

- [Baseline comparison](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/comparison.py): `src/evalforge/comparison.py`
- [Regression gates](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/comparison.py): `src/evalforge/comparison.py`

### Records and reports

- [Config and run IO](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/io.py): `src/evalforge/io.py`
- [Run records](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/runner.py): `src/evalforge/runner.py`
- [Run reports](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/reporting.py): `src/evalforge/reporting.py`
- [Comparison reports](https://github.com/sunny-arch-82/ai-evaluation-and-regression-testing-framework/blob/main/src/evalforge/reporting.py): `src/evalforge/reporting.py`

### Other components

- **Evaluator**
- **User Python system**
- **HTTP evaluation endpoint**

## Connections

- Evaluator → CLI commands: invokes (evidence: `README.md`)
- Run configuration → CLI commands: configures (evidence: `README.md`)
- Golden cases → Config and run IO: loaded as (evidence: `README.md`)
- CLI commands → Sequential runner: starts run (evidence: `src/evalforge/cli.py`)
- Sequential runner → Config and run IO: loads and saves (evidence: `src/evalforge/runner.py`)
- Config and run IO → Case and output contracts: validates (evidence: `src/evalforge/io.py`)
- Sequential runner → Case and output contracts: validates output (evidence: `src/evalforge/runner.py`)
- Sequential runner → Target adapters: builds and calls (evidence: `src/evalforge/runner.py`)
- Target adapters → Demo behaviors: calls demo (evidence: `src/evalforge/targets.py`)
- Target adapters → Python target: dispatches Python (evidence: `README.md`)
- Target adapters → Saved-output target: dispatches offline (evidence: `README.md`)
- Target adapters → HTTP target: dispatches HTTP (evidence: `README.md`)
- Python target → User Python system: invokes (evidence: `README.md`)
- HTTP target → HTTP evaluation endpoint: posts cases (evidence: `README.md`)
- Saved-output target → Saved responses: reads (evidence: `README.md`)
- Sequential runner → Metrics and aggregation: scores and aggregates (evidence: `src/evalforge/runner.py`)
- Sequential runner → Run records: writes (evidence: `src/evalforge/runner.py`)
- Sequential runner → Run reports: generates (evidence: `src/evalforge/runner.py`)
- CLI commands → Run reports: regenerates (evidence: `src/evalforge/cli.py`)
- CLI commands → Baseline comparison: starts comparison (evidence: `src/evalforge/cli.py`)
- Baseline comparison → Config and run IO: loads runs (evidence: `src/evalforge/comparison.py`)
- Baseline comparison → Case and output contracts: validates gate config (evidence: `src/evalforge/comparison.py`)
- Baseline comparison → Regression gates: evaluates (evidence: `src/evalforge/comparison.py`)
- CLI commands → Comparison reports: writes comparison (evidence: `src/evalforge/cli.py`)