How darthvader19/agent_eval works
This is an agent evaluation and benchmarking framework. Its main workflow loads benchmark tasks and experiment settings, runs selected agent architectures under shared budgets and tools, grades answers, then persists repeat-level results for analysis. The map includes documented reflection and optional framework comparisons. Some components are represented by directory paths because their implementation was not sampled; links to them are included only where the README or SOURCE INDEX supports the relationship.
Main entry points: runner.py Evaluation runner · toolbox.py Shared tool interface · plan_execute_agent.py Plan and execute · langgraph_agent.py LangGraph agent
Components 26
Evaluation workflow
- Evaluation runner
harness/runner.py - Task suites
benchmarks - Experiment settings
configs/experiments
Agent architectures
- Agent contract
agents/base.py - ReAct agent
agents/react_agent.py - Plan and execute
agents/plan_execute_agent.py - LangGraph agent
agents/langgraph_agent.py - Direct baseline
agents/direct_baseline.py - Framework adapters
frameworks - Step execution
agents/runtime.py
Shared tools
- Shared tool interface
tools/toolbox.py - Calculator
tools/calculator.py - Python sandbox
tools/python_sandbox.py - Snapshot search
tools/search.py - Local retrieval
tools/retriever.py - Fault injection
tools/fault_injection.py - Benchmark corpus
benchmarks/corpus
Scoring and analysis
- Answer grading
harness/judge.py - Paired reflection
harness/reflection.py - Run metrics
harness/metrics.py - Statistical summaries
harness/stats.py - Result analysis
analysis/aggregate_results.py
Models and persistence
- Model client
harness/model.py - Chat completions API
- Run results
results
Other components
- Evaluator
Connections 31
- Evaluator → Evaluation runner: starts run
README.md - Evaluation runner → Task suites: loads tasks
README.md - Evaluation runner → Experiment settings: uses settings
README.md - Evaluation runner → ReAct agent: runs agent
harness/runner.py - Evaluation runner → Plan and execute: runs agent
harness/runner.py - Evaluation runner → LangGraph agent: runs agent
harness/runner.py - Evaluation runner → Direct baseline: runs agent
harness/runner.py - Evaluation runner → Framework adapters: runs adapters
README.md - ReAct agent → Agent contract: implements
agents/react_agent.py - Plan and execute → Step execution: executes steps
agents/plan_execute_agent.py - LangGraph agent → Step execution: executes steps
agents/langgraph_agent.py - Step execution → Shared tool interface: invokes tools
agents/runtime.py - Plan and execute → Shared tool interface: uses tools
agents/plan_execute_agent.py - LangGraph agent → Shared tool interface: uses tools
agents/langgraph_agent.py - Framework adapters → Shared tool interface: shares interface
frameworks/crewai_runner.py - Shared tool interface → Calculator: dispatches to
tools/toolbox.py - Shared tool interface → Python sandbox: dispatches to
tools/toolbox.py - Shared tool interface → Snapshot search: dispatches to
tools/toolbox.py - Shared tool interface → Local retrieval: dispatches to
tools/toolbox.py - Shared tool interface → Fault injection: applies faults
tools/toolbox.py - Snapshot search → Benchmark corpus: reads snapshots
tools/search.py - Local retrieval → Benchmark corpus: reads documents
tools/retriever.py - Evaluation runner → Answer grading: grades answers
harness/runner.py - Evaluation runner → Paired reflection: runs reflection
harness/runner.py - Evaluation runner → Run metrics: aggregates metrics
harness/runner.py - Run metrics → Statistical summaries: uses statistics
harness/metrics.py - Result analysis → Run results: analyzes runs
README.md - Evaluation runner → Run results: persists results
README.md - Agent contract → Model client: creates client
agents/base.py - Answer grading → Model client: uses for LLM grading
harness/judge.py - Model client → Chat completions API: sends completions
harness/model.py