How alexcpn/eval-driven-development works
A financial-market research agent is driven by a versioned prompt/model/evaluation contract. The main workflow loads a case, builds an agent with SearchAPI MCP tools, runs it against the configured model, and prints the response and usage. Separate live and stored-trace evaluation paths assess behavior; observability exports traces, logs, and metrics to a local dashboard stack. Links for unsampled evaluation runners and infrastructure are based on README documentation, not inspected implementation.
Main entry points: main.py CLI Runner · observability.py OpenTelemetry Export · tempo.yml Tempo · financial_agent.py Financial Agent
Components 18
Agent Runtime
- CLI Runner
main.py - Financial Agent
financial_agent.py - OpenAI GPT
- SearchAPI MCP
Behavior Contract
- Eval Case Loader
eval_cases.py - Prompt Contract
prompt_eval.json
Evaluation
- Live Repeated Evals
tests/unittest_eval_agent.py - Trace Regression
tests/regression_eval_traces.py - LLM Judge Regression
tests/regression_llm_judge_traces.py - Tool Call Evaluator
eval_evaluators.py - Eval Reports
tests/reports/regression_llm_judge_traces.json
Observability
- OpenTelemetry Export
observability.py - Prometheus
infra/prometheus.yml - Tempo
infra/tempo.yml - Loki
infra/loki-config.yml - SeaweedFS
infra/docker-compose.yml - Grafana
infra/grafana/provisioning/dashboards/financial-agent-observability.json
Other components
- User
Connections 22
- User → CLI Runner: runs
README.md - CLI Runner → Eval Case Loader: loads case
main.py - Eval Case Loader → Prompt Contract: reads
eval_cases.py - CLI Runner → Financial Agent: builds and runs
main.py - Financial Agent → OpenAI GPT: requests completion
README.md - Financial Agent → SearchAPI MCP: calls tools
financial_agent.py - CLI Runner → OpenTelemetry Export: configures and flushes
main.py - CLI Runner → OpenTelemetry Export: records usage
main.py - OpenTelemetry Export → Tempo: exports traces
README.md - OpenTelemetry Export → Loki: exports logs
README.md - OpenTelemetry Export → Prometheus: serves metrics
README.md - Tempo → SeaweedFS: stores trace blocks
README.md - Prometheus → Grafana: visualized by
README.md - Tempo → Grafana: visualized by
README.md - Loki → Grafana: visualized by
README.md - Live Repeated Evals → Eval Case Loader: uses cases
README.md - Live Repeated Evals → Financial Agent: runs cases
README.md - Trace Regression → Tempo: evaluates stored traces
README.md - LLM Judge Regression → Tempo: evaluates stored traces
README.md - Trace Regression → Tool Call Evaluator: applies checks
- Trace Regression → Eval Reports: produces report
README.md - LLM Judge Regression → Eval Reports: produces report
README.md