How sunny-arch-82/ai-evaluation-and-regression-testing-framework works
EvalForge is a local evaluation and regression framework, not a chatbot: users provide labeled cases and a YAML run configuration, then select a target adapter. The runner executes cases, validates target outputs, scores them, and saves immutable run records and reports. Users can then compare compatible runs and apply fail-closed gates. The graph also preserves citation, retrieval, structured-output, tool-trace, cost, latency, and offline/HTTP/Python adapter capabilities. Relationships are based on the README and shown source; unsampled subsystems are included only where documented.
Main entry points: runner.py Sequential runner · cli.py CLI commands · targets.py Target adapters · comparison.py Baseline comparison
Components 21
Contracts and inputs
- Run configuration
configs - Golden cases
data/sample - Case and output contracts
src/evalforge/schemas.py - Saved responses
data/sample/saved_outputs.jsonl
Target adapters
- Target adapters
src/evalforge/targets.py - Demo behaviors
src/evalforge/demo.py - Python target
src/evalforge/targets.py - Saved-output target
src/evalforge/targets.py - HTTP target
src/evalforge/targets.py
Run and scoring
- CLI commands
src/evalforge/cli.py - Sequential runner
src/evalforge/runner.py - Metrics and aggregation
src/evalforge/metrics.py
Comparison and gates
- Baseline comparison
src/evalforge/comparison.py - Regression gates
src/evalforge/comparison.py
Records and reports
- Config and run IO
src/evalforge/io.py - Run records
src/evalforge/runner.py - Run reports
src/evalforge/reporting.py - Comparison reports
src/evalforge/reporting.py
Other components
- Evaluator
- User Python system
- HTTP evaluation endpoint
Connections 24
- Evaluator → CLI commands: invokes
README.md - Run configuration → CLI commands: configures
README.md - Golden cases → Config and run IO: loaded as
README.md - CLI commands → Sequential runner: starts run
src/evalforge/cli.py - Sequential runner → Config and run IO: loads and saves
src/evalforge/runner.py - Config and run IO → Case and output contracts: validates
src/evalforge/io.py - Sequential runner → Case and output contracts: validates output
src/evalforge/runner.py - Sequential runner → Target adapters: builds and calls
src/evalforge/runner.py - Target adapters → Demo behaviors: calls demo
src/evalforge/targets.py - Target adapters → Python target: dispatches Python
README.md - Target adapters → Saved-output target: dispatches offline
README.md - Target adapters → HTTP target: dispatches HTTP
README.md - Python target → User Python system: invokes
README.md - HTTP target → HTTP evaluation endpoint: posts cases
README.md - Saved-output target → Saved responses: reads
README.md - Sequential runner → Metrics and aggregation: scores and aggregates
src/evalforge/runner.py - Sequential runner → Run records: writes
src/evalforge/runner.py - Sequential runner → Run reports: generates
src/evalforge/runner.py - CLI commands → Run reports: regenerates
src/evalforge/cli.py - CLI commands → Baseline comparison: starts comparison
src/evalforge/cli.py - Baseline comparison → Config and run IO: loads runs
src/evalforge/comparison.py - Baseline comparison → Case and output contracts: validates gate config
src/evalforge/comparison.py - Baseline comparison → Regression gates: evaluates
src/evalforge/comparison.py - CLI commands → Comparison reports: writes comparison
src/evalforge/cli.py