# alexcpn/eval-driven-development architecture

> Architecture diagram of the GitHub repository alexcpn/eval-driven-development, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alexcpn/eval-driven-development
- Repository: https://github.com/alexcpn/eval-driven-development

## Overview

A financial-market research agent is driven by a versioned prompt/model/evaluation contract. The main workflow loads a case, builds an agent with SearchAPI MCP tools, runs it against the configured model, and prints the response and usage. Separate live and stored-trace evaluation paths assess behavior; observability exports traces, logs, and metrics to a local dashboard stack. Links for unsampled evaluation runners and infrastructure are based on README documentation, not inspected implementation.

Main entry points: [`main.py`](https://github.com/alexcpn/eval-driven-development/blob/main/main.py) (CLI Runner), [`observability.py`](https://github.com/alexcpn/eval-driven-development/blob/main/observability.py) (OpenTelemetry Export), [`infra/tempo.yml`](https://github.com/alexcpn/eval-driven-development/blob/main/infra/tempo.yml) (Tempo), [`financial_agent.py`](https://github.com/alexcpn/eval-driven-development/blob/main/financial_agent.py) (Financial Agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Agent Runtime"]
  node_main["CLI Runner<br/>[main.py]"]
  node_agent["Financial Agent<br/>[financial_agent.py]"]
  node_model["OpenAI GPT"]
  node_search["SearchAPI MCP"]
end

subgraph group_contract["Behavior Contract"]
  node_case["Eval Case Loader<br/>[eval_cases.py]"]
  node_contractfile["Prompt Contract<br/>[prompt_eval.json]"]
end

subgraph group_evaluation["Evaluation"]
  node_live["Live Repeated Evals"]
  node_traceeval["Trace Regression"]
  node_judge["LLM Judge Regression"]
  node_evaluator["Tool Call Evaluator<br/>[eval_evaluators.py]"]
  node_reports["Eval Reports"]
end

subgraph group_telemetry["Observability"]
  node_otel["OpenTelemetry Export<br/>[observability.py]"]
  node_prom[("Prometheus<br/>[prometheus.yml]")]
  node_tempo[("Tempo<br/>[tempo.yml]")]
  node_loki[("Loki<br/>[loki-config.yml]")]
  node_seaweed[("SeaweedFS<br/>[docker-compose.yml]")]
  node_grafana["Grafana"]
end

node_user(("User"))

node_user -->|"runs"| node_main
node_main -->|"loads case"| node_case
node_case -->|"reads"| node_contractfile
node_main -->|"builds and runs"| node_agent
node_agent -->|"requests completion"| node_model
node_agent -->|"calls tools"| node_search
node_main -->|"configures and flushes"| node_otel
node_main -->|"records usage"| node_otel
node_otel -->|"exports traces"| node_tempo
node_otel -->|"exports logs"| node_loki
node_otel -->|"serves metrics"| node_prom
node_tempo -->|"stores trace blocks"| node_seaweed
node_prom -->|"visualized by"| node_grafana
node_tempo -->|"visualized by"| node_grafana
node_loki -->|"visualized by"| node_grafana
node_live -->|"uses cases"| node_case
node_live -->|"runs cases"| node_agent
node_traceeval -->|"evaluates stored traces"| node_tempo
node_judge -->|"evaluates stored traces"| node_tempo
node_traceeval -.->|"applies checks"| node_evaluator
node_traceeval -->|"produces report"| node_reports
node_judge -->|"produces report"| node_reports

click node_main "https://github.com/alexcpn/eval-driven-development/blob/main/main.py"
click node_agent "https://github.com/alexcpn/eval-driven-development/blob/main/financial_agent.py"
click node_case "https://github.com/alexcpn/eval-driven-development/blob/main/eval_cases.py"
click node_contractfile "https://github.com/alexcpn/eval-driven-development/blob/main/prompt_eval.json"
click node_live "https://github.com/alexcpn/eval-driven-development/blob/main/tests/unittest_eval_agent.py"
click node_traceeval "https://github.com/alexcpn/eval-driven-development/blob/main/tests/regression_eval_traces.py"
click node_judge "https://github.com/alexcpn/eval-driven-development/blob/main/tests/regression_llm_judge_traces.py"
click node_evaluator "https://github.com/alexcpn/eval-driven-development/blob/main/eval_evaluators.py"
click node_reports "https://github.com/alexcpn/eval-driven-development/blob/main/tests/reports/regression_llm_judge_traces.json"
click node_otel "https://github.com/alexcpn/eval-driven-development/blob/main/observability.py"
click node_prom "https://github.com/alexcpn/eval-driven-development/blob/main/infra/prometheus.yml"
click node_tempo "https://github.com/alexcpn/eval-driven-development/blob/main/infra/tempo.yml"
click node_loki "https://github.com/alexcpn/eval-driven-development/blob/main/infra/loki-config.yml"
click node_seaweed "https://github.com/alexcpn/eval-driven-development/blob/main/infra/docker-compose.yml"
click node_grafana "https://github.com/alexcpn/eval-driven-development/blob/main/infra/grafana/provisioning/dashboards/financial-agent-observability.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_agent,node_model,node_search,node_user toneBlue
class node_case,node_contractfile toneAmber
class node_live,node_traceeval,node_judge,node_evaluator,node_reports toneMint
class node_otel,node_prom,node_tempo,node_loki,node_seaweed,node_grafana toneRose
```

## Components

### Agent Runtime

- [CLI Runner](https://github.com/alexcpn/eval-driven-development/blob/main/main.py): `main.py`
- [Financial Agent](https://github.com/alexcpn/eval-driven-development/blob/main/financial_agent.py): `financial_agent.py`
- **OpenAI GPT**
- **SearchAPI MCP**

### Behavior Contract

- [Eval Case Loader](https://github.com/alexcpn/eval-driven-development/blob/main/eval_cases.py): `eval_cases.py`
- [Prompt Contract](https://github.com/alexcpn/eval-driven-development/blob/main/prompt_eval.json): `prompt_eval.json`

### Evaluation

- [Live Repeated Evals](https://github.com/alexcpn/eval-driven-development/blob/main/tests/unittest_eval_agent.py): `tests/unittest_eval_agent.py`
- [Trace Regression](https://github.com/alexcpn/eval-driven-development/blob/main/tests/regression_eval_traces.py): `tests/regression_eval_traces.py`
- [LLM Judge Regression](https://github.com/alexcpn/eval-driven-development/blob/main/tests/regression_llm_judge_traces.py): `tests/regression_llm_judge_traces.py`
- [Tool Call Evaluator](https://github.com/alexcpn/eval-driven-development/blob/main/eval_evaluators.py): `eval_evaluators.py`
- [Eval Reports](https://github.com/alexcpn/eval-driven-development/blob/main/tests/reports/regression_llm_judge_traces.json): `tests/reports/regression_llm_judge_traces.json`

### Observability

- [OpenTelemetry Export](https://github.com/alexcpn/eval-driven-development/blob/main/observability.py): `observability.py`
- [Prometheus](https://github.com/alexcpn/eval-driven-development/blob/main/infra/prometheus.yml): `infra/prometheus.yml`
- [Tempo](https://github.com/alexcpn/eval-driven-development/blob/main/infra/tempo.yml): `infra/tempo.yml`
- [Loki](https://github.com/alexcpn/eval-driven-development/blob/main/infra/loki-config.yml): `infra/loki-config.yml`
- [SeaweedFS](https://github.com/alexcpn/eval-driven-development/blob/main/infra/docker-compose.yml): `infra/docker-compose.yml`
- [Grafana](https://github.com/alexcpn/eval-driven-development/blob/main/infra/grafana/provisioning/dashboards/financial-agent-observability.json): `infra/grafana/provisioning/dashboards/financial-agent-observability.json`

### Other components

- **User**

## Connections

- User → CLI Runner: runs (evidence: `README.md`)
- CLI Runner → Eval Case Loader: loads case (evidence: `main.py`)
- Eval Case Loader → Prompt Contract: reads (evidence: `eval_cases.py`)
- CLI Runner → Financial Agent: builds and runs (evidence: `main.py`)
- Financial Agent → OpenAI GPT: requests completion (evidence: `README.md`)
- Financial Agent → SearchAPI MCP: calls tools (evidence: `financial_agent.py`)
- CLI Runner → OpenTelemetry Export: configures and flushes (evidence: `main.py`)
- CLI Runner → OpenTelemetry Export: records usage (evidence: `main.py`)
- OpenTelemetry Export → Tempo: exports traces (evidence: `README.md`)
- OpenTelemetry Export → Loki: exports logs (evidence: `README.md`)
- OpenTelemetry Export → Prometheus: serves metrics (evidence: `README.md`)
- Tempo → SeaweedFS: stores trace blocks (evidence: `README.md`)
- Prometheus → Grafana: visualized by (evidence: `README.md`)
- Tempo → Grafana: visualized by (evidence: `README.md`)
- Loki → Grafana: visualized by (evidence: `README.md`)
- Live Repeated Evals → Eval Case Loader: uses cases (evidence: `README.md`)
- Live Repeated Evals → Financial Agent: runs cases (evidence: `README.md`)
- Trace Regression → Tempo: evaluates stored traces (evidence: `README.md`)
- LLM Judge Regression → Tempo: evaluates stored traces (evidence: `README.md`)
- Trace Regression → Tool Call Evaluator: applies checks
- Trace Regression → Eval Reports: produces report (evidence: `README.md`)
- LLM Judge Regression → Eval Reports: produces report (evidence: `README.md`)
