How guybass/agent-eval-flow works
Agent Eval Flow evaluates agent runs and configuration changes from retained or newly captured evidence. The principal workflow is to define candidates and checks, run agents or import captures through adapters, evaluate evidence, inspect HTML reports, and compare or select candidate results. Separate assessment workflows cover configuration and observed runtime behavior; Toolscore adds optional tool-call diagnostics. The graph preserves these documented capabilities. Some implementation paths are represented at their exact module paths; relationships not shown in the excerpts or source references are omitted.
Main entry points: assessment.py Configuration assessment · runner.py Run dispatcher · query.py Result queries · engine.py Evaluation engine
Components 20
Workflow
- Evaluation pipeline
src/agent_eval_flow/pipeline/api.py - Configuration assessment
src/agent_eval_flow/pipeline/assessment.py
Capture and adapters
- Run dispatcher
src/agent_eval_flow/execution/runner.py - Capture receipts
src/agent_eval_flow/execution/capture.py - Runtime adapters
src/agent_eval_flow/adapters - Adapter utilities
src/agent_eval_flow/adapters/common.py
Evidence model
- Run records
src/agent_eval_flow/objects/records.py - Assessment records
src/agent_eval_flow/objects/assessment.py - Assessment validation
src/agent_eval_flow/objects/assessment_validation.py - Stable identities
src/agent_eval_flow/objects/identity.py
Evaluation and results
- Evaluation engine
src/agent_eval_flow/evaluation/engine.py - Runtime checks
src/agent_eval_flow/evaluation/runtime_checks.py - Toolscore evaluator
src/agent_eval_flow/evaluation/toolscore.py - Result queries
src/agent_eval_flow/results/query.py - Candidate selection
src/agent_eval_flow/results/assessment_selection.py
Reports and storage
- HTML reports
src/agent_eval_flow/reporting/html.py - Toolscore report
src/agent_eval_flow/reporting/toolscore.py - Evidence artifacts
src/agent_eval_flow/storage/artifacts.py - Result manifests
src/agent_eval_flow/storage/manifests.py
Other components
- Developer
Connections 22
- Developer → Evaluation pipeline: defines evaluation
README.md - Developer → Configuration assessment: assesses configuration
README.md - Evaluation pipeline → Run dispatcher: runs agents
src/agent_eval_flow/pipeline/api.py - Evaluation pipeline → Evaluation engine: evaluates captures
src/agent_eval_flow/pipeline/api.py - Configuration assessment → Run dispatcher: captures behavior
src/agent_eval_flow/pipeline/assessment.py - Configuration assessment → Evaluation engine: evaluates checks
src/agent_eval_flow/pipeline/assessment.py - Run dispatcher → Capture receipts: records receipts
src/agent_eval_flow/execution/runner.py - Run dispatcher → Runtime adapters: dispatches runs
README.md - Runtime adapters → Adapter utilities: uses utilities
src/agent_eval_flow/adapters/openkritt.py - Adapter utilities → Evidence artifacts: stores evidence
src/agent_eval_flow/adapters/common.py - Run dispatcher → Run records: returns run records
src/agent_eval_flow/execution/runner.py - Run records → Stable identities: fingerprints records
src/agent_eval_flow/objects/records.py - Assessment records → Assessment validation: validates records
src/agent_eval_flow/objects/assessment.py - Configuration assessment → Runtime checks: checks behavior
src/agent_eval_flow/pipeline/assessment.py - Configuration assessment → Candidate selection: selects candidates
src/agent_eval_flow/pipeline/assessment.py - Evaluation engine → Result queries: produces queryable results
src/agent_eval_flow/evaluation/engine.py - Toolscore evaluator → Toolscore report: creates diagnostics
src/agent_eval_flow/evaluation/toolscore.py - Result queries → HTML reports: supplies report data
src/agent_eval_flow/reporting/html.py - Result queries → Candidate selection: supports selection
src/agent_eval_flow/results/assessment_selection.py - HTML reports → Developer: presents findings
README.md - Evaluation engine → Evidence artifacts: reads evidence
- Result queries → Result manifests: loads saved results
src/agent_eval_flow/storage/manifests.py