# evolving-foundry/ember-evaluator architecture

> Architecture diagram of the GitHub repository evolving-foundry/ember-evaluator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/evolving-foundry/ember-evaluator
- Repository: https://github.com/evolving-foundry/ember-evaluator

## Overview

Ember Evaluator is a plugin-based framework for evaluating agent harnesses, inference engines, and kernels. Its main workflow starts with a YAML configuration, resolves selected benchmark and runtime components, prepares tasks and environments, executes solver and verification stages, then stores results for metrics and reports. Local, Docker, and Ray execution are supported; resume, retries, diagnostics, and warmup extend that workflow. The repository also includes the Tracer plugin for trajectory capture, visualization, analysis, and feedback, plus AgentX benchmarking. Arrows use shown source references or README statements; unsampled subsystem links are omitted or marked unverified.

Main entry points: [`src/core/execution/engine.ts`](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/engine.ts) (Run engine), [`plugins/tracer/collector/routes-ingest.ts`](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/routes-ingest.ts) (Collector routes), [`src/cli/main.ts`](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/cli/main.ts) (CLI commands), [`src/core/execution/run-task.ts`](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/run-task.ts) (Task execution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User entry and configuration"]
  node_cli["CLI commands<br/>[main.ts]"]
  node_config["Configuration<br/>[resolve.ts]"]
  node_pluginreg["Plugin registry<br/>[registry.ts]"]
end

subgraph group_evaluation["Evaluation workflow"]
  node_run["Run application<br/>[run.ts]"]
  node_engine["Run engine<br/>[engine.ts]"]
  node_benchmark["Benchmark and tasks<br/>[benchmarks.ts]"]
  node_selection["Task selection<br/>[task-selection.ts]"]
  node_preparation["Task preparation"]
  node_execution["Task execution<br/>[run-task.ts]"]
  node_errors["Failure diagnostics<br/>[classify.ts]"]
  node_contracts["Evaluation contracts<br/>[result.ts]"]
end

subgraph group_runtime["Execution runtimes"]
  node_environment["Local and Docker environments<br/>[session.ts]"]
  node_solvers["Agent and model runtimes<br/>[simpleagent.ts]"]
  node_ray["Ray workers and jobs<br/>[client.ts]"]
  node_agentx["AgentX performance benchmark<br/>[index.ts]"]
end

subgraph group_results["Results and reporting"]
  node_store[("Run result store<br/>[local.ts]")]
  node_metrics["Metrics<br/>[aggregate.ts]"]
  node_reports["Reports and snapshots<br/>[index.ts]"]
end

subgraph group_tracer["Tracer observability"]
  node_tracerplugin["Tracer plugin<br/>[index.ts]"]
  node_traceclient["Trace client<br/>[client.ts]"]
  node_traceingest["Trace ingestion<br/>[ingest.ts]"]
  node_traceroutes["Collector routes<br/>[routes-ingest.ts]"]
  node_tracequery["Query and analysis routes<br/>[routes-query.ts]"]
  node_traceanalysis["Trajectory analysis<br/>[llm-analysis.ts]"]
  node_tracestore[("Trace store<br/>[trace-store.ts]")]
  node_traceview["Trajectory viewer<br/>[app.js]"]
end

node_user(("Evaluator user"))

node_user -->|"runs commands"| node_cli
node_cli -->|"loads settings"| node_config
node_cli -->|"starts evaluation"| node_run
node_cli -->|"requests reports"| node_reports
node_run -.->|"runs engine"| node_engine
node_config -.->|"selects components"| node_pluginreg
node_engine -->|"resolves source"| node_benchmark
node_engine -->|"enumerates tasks"| node_selection
node_engine -.->|"prepares tasks"| node_preparation
node_engine -.->|"dispatches tasks"| node_execution
node_preparation -->|"classifies failures"| node_errors
node_execution -.->|"uses environment"| node_environment
node_execution -.->|"runs solver"| node_solvers
node_engine -.->|"persists run"| node_store
node_execution -.->|"produces results"| node_contracts
node_engine -.->|"supports Ray jobs"| node_ray
node_store -.->|"provides saved results"| node_metrics
node_reports -.->|"reads run data"| node_store
node_tracerplugin -->|"registers client"| node_traceclient
node_traceclient -.->|"sends traces"| node_traceroutes
node_traceroutes -->|"queues traces"| node_traceingest
node_traceroutes -->|"exports analysis"| node_traceanalysis
node_tracequery -->|"requests analysis"| node_traceanalysis
node_traceroutes -.->|"records traces"| node_tracestore
node_tracequery -.->|"queries traces"| node_tracestore
node_traceview -.->|"views trajectories"| node_traceroutes
node_agentx -.->|"replays model requests"| node_solvers

click node_cli "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/cli/main.ts"
click node_config "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/config/resolve.ts"
click node_pluginreg "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/plugins/registry.ts"
click node_run "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/run.ts"
click node_engine "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/engine.ts"
click node_benchmark "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/benchmarks.ts"
click node_selection "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/selection/task-selection.ts"
click node_preparation "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/schedulers/local/preparation-pipeline.ts"
click node_execution "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/run-task.ts"
click node_errors "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/errors/classify.ts"
click node_environment "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/environments/docker/session.ts"
click node_solvers "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/solvers/simpleagent.ts"
click node_ray "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/infrastructure/ray/client.ts"
click node_contracts "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/contracts/result.ts"
click node_store "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/stores/local.ts"
click node_metrics "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/metrics/aggregate.ts"
click node_reports "https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/report/index.ts"
click node_tracerplugin "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/index.ts"
click node_traceclient "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/client.ts"
click node_traceingest "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/ingest.ts"
click node_traceroutes "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/routes-ingest.ts"
click node_tracequery "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/routes-query.ts"
click node_traceanalysis "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/llm-analysis.ts"
click node_tracestore "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/trace-store.ts"
click node_traceview "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/view/app.js"
click node_agentx "https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/aisbench/index.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_pluginreg,node_user toneBlue
class node_run,node_engine,node_benchmark,node_selection,node_preparation,node_execution,node_errors,node_contracts toneAmber
class node_environment,node_solvers,node_ray,node_agentx toneMint
class node_store,node_metrics,node_reports toneRose
class node_tracerplugin,node_traceclient,node_traceingest,node_traceroutes,node_tracequery,node_traceanalysis,node_tracestore,node_traceview toneIndigo
```

## Components

### User entry and configuration

- [CLI commands](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/cli/main.ts): `src/cli/main.ts`
- [Configuration](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/config/resolve.ts): `src/config/resolve.ts`
- [Plugin registry](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/plugins/registry.ts): `src/application/plugins/registry.ts`

### Evaluation workflow

- [Run application](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/run.ts): `src/application/run.ts`
- [Run engine](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/engine.ts): `src/core/execution/engine.ts`
- [Benchmark and tasks](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/benchmarks.ts): `src/application/benchmarks.ts`
- [Task selection](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/selection/task-selection.ts): `src/core/selection/task-selection.ts`
- [Task preparation](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/schedulers/local/preparation-pipeline.ts): `src/plugins/schedulers/local/preparation-pipeline.ts`
- [Task execution](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/execution/run-task.ts): `src/core/execution/run-task.ts`
- [Failure diagnostics](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/errors/classify.ts): `src/core/errors/classify.ts`
- [Evaluation contracts](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/core/contracts/result.ts): `src/core/contracts/result.ts`

### Execution runtimes

- [Local and Docker environments](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/environments/docker/session.ts): `src/plugins/environments/docker/session.ts`
- [Agent and model runtimes](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/solvers/simpleagent.ts): `src/plugins/solvers/simpleagent.ts`
- [Ray workers and jobs](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/infrastructure/ray/client.ts): `src/infrastructure/ray/client.ts`
- [AgentX performance benchmark](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/aisbench/index.ts): `plugins/aisbench/index.ts`

### Results and reporting

- [Run result store](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/stores/local.ts): `src/plugins/stores/local.ts`
- [Metrics](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/plugins/metrics/aggregate.ts): `src/plugins/metrics/aggregate.ts`
- [Reports and snapshots](https://github.com/evolving-foundry/ember-evaluator/blob/master/src/application/report/index.ts): `src/application/report/index.ts`

### Tracer observability

- [Tracer plugin](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/index.ts): `plugins/tracer/index.ts`
- [Trace client](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/client.ts): `plugins/tracer/client.ts`
- [Trace ingestion](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/ingest.ts): `plugins/tracer/collector/ingest.ts`
- [Collector routes](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/routes-ingest.ts): `plugins/tracer/collector/routes-ingest.ts`
- [Query and analysis routes](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/routes-query.ts): `plugins/tracer/collector/routes-query.ts`
- [Trajectory analysis](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/llm-analysis.ts): `plugins/tracer/collector/llm-analysis.ts`
- [Trace store](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/trace-store.ts): `plugins/tracer/collector/trace-store.ts`
- [Trajectory viewer](https://github.com/evolving-foundry/ember-evaluator/blob/master/plugins/tracer/collector/view/app.js): `plugins/tracer/collector/view/app.js`

### Other components

- **Evaluator user**

## Connections

- Evaluator user → CLI commands: runs commands (evidence: `README.md`)
- CLI commands → Configuration: loads settings (evidence: `src/cli/main.ts`)
- CLI commands → Run application: starts evaluation (evidence: `src/cli/main.ts`)
- CLI commands → Reports and snapshots: requests reports (evidence: `src/cli/main.ts`)
- Run application → Run engine: runs engine
- Configuration → Plugin registry: selects components
- Run engine → Benchmark and tasks: resolves source (evidence: `src/core/execution/engine.ts`)
- Run engine → Task selection: enumerates tasks (evidence: `src/core/execution/engine.ts`)
- Run engine → Task preparation: prepares tasks
- Run engine → Task execution: dispatches tasks (evidence: `src/core/execution/engine.ts`)
- Task preparation → Failure diagnostics: classifies failures (evidence: `src/plugins/schedulers/local/preparation-pipeline.ts`)
- Task execution → Local and Docker environments: uses environment
- Task execution → Agent and model runtimes: runs solver
- Run engine → Run result store: persists run
- Task execution → Evaluation contracts: produces results
- Run engine → Ray workers and jobs: supports Ray jobs
- Run result store → Metrics: provides saved results
- Reports and snapshots → Run result store: reads run data
- Tracer plugin → Trace client: registers client
- Trace client → Collector routes: sends traces
- Collector routes → Trace ingestion: queues traces (evidence: `plugins/tracer/collector/routes-ingest.ts`)
- Collector routes → Trajectory analysis: exports analysis (evidence: `plugins/tracer/collector/routes-ingest.ts`)
- Query and analysis routes → Trajectory analysis: requests analysis (evidence: `plugins/tracer/collector/routes-query.ts`)
- Collector routes → Trace store: records traces
- Query and analysis routes → Trace store: queries traces
- Trajectory viewer → Collector routes: views trajectories
- AgentX performance benchmark → Agent and model runtimes: replays model requests (evidence: `README.md`)