evolving-foundry/ember-evaluator

How evolving-foundry/ember-evaluator works

Ember Evaluator is a plugin-based framework for evaluating agent harnesses, inference engines, and kernels. Its main workflow starts with a YAML configuration, resolves selected benchmark and runtime components, prepares tasks and environments, executes solver and verification stages, then stores results for metrics and reports. Local, Docker, and Ray execution are supported; resume, retries, diagnostics, and warmup extend that workflow. The repository also includes the Tracer plugin for trajectory capture, visualization, analysis, and feedback, plus AgentX benchmarking. Arrows use shown source references or README statements; unsampled subsystem links are omitted or marked unverified.

Main entry points: engine.ts Run engine · routes-ingest.ts Collector routes · main.ts CLI commands · run-task.ts Task execution

Components 27

User entry and configuration

CLI commands src/cli/main.ts
Configuration src/config/resolve.ts
Plugin registry src/application/plugins/registry.ts

Evaluation workflow

Run application src/application/run.ts
Run engine src/core/execution/engine.ts
Benchmark and tasks src/application/benchmarks.ts
Task selection src/core/selection/task-selection.ts
Task preparation src/plugins/schedulers/local/preparation-pipeline.ts
Task execution src/core/execution/run-task.ts
Failure diagnostics src/core/errors/classify.ts
Evaluation contracts src/core/contracts/result.ts

Execution runtimes

Local and Docker environments src/plugins/environments/docker/session.ts
Agent and model runtimes src/plugins/solvers/simpleagent.ts
Ray workers and jobs src/infrastructure/ray/client.ts
AgentX performance benchmark plugins/aisbench/index.ts

Results and reporting

Run result store src/plugins/stores/local.ts
Metrics src/plugins/metrics/aggregate.ts
Reports and snapshots src/application/report/index.ts

Tracer observability

Tracer plugin plugins/tracer/index.ts
Trace client plugins/tracer/client.ts
Trace ingestion plugins/tracer/collector/ingest.ts
Collector routes plugins/tracer/collector/routes-ingest.ts
Query and analysis routes plugins/tracer/collector/routes-query.ts
Trajectory analysis plugins/tracer/collector/llm-analysis.ts
Trace store plugins/tracer/collector/trace-store.ts
Trajectory viewer plugins/tracer/collector/view/app.js

Other components

Evaluator user

Connections 27

Evaluator user → CLI commands: runs commands README.md
CLI commands → Configuration: loads settings src/cli/main.ts
CLI commands → Run application: starts evaluation src/cli/main.ts
CLI commands → Reports and snapshots: requests reports src/cli/main.ts
Run application → Run engine: runs engine
Configuration → Plugin registry: selects components
Run engine → Benchmark and tasks: resolves source src/core/execution/engine.ts
Run engine → Task selection: enumerates tasks src/core/execution/engine.ts
Run engine → Task preparation: prepares tasks
Run engine → Task execution: dispatches tasks src/core/execution/engine.ts
Task preparation → Failure diagnostics: classifies failures src/plugins/schedulers/local/preparation-pipeline.ts
Task execution → Local and Docker environments: uses environment
Task execution → Agent and model runtimes: runs solver
Run engine → Run result store: persists run
Task execution → Evaluation contracts: produces results
Run engine → Ray workers and jobs: supports Ray jobs
Run result store → Metrics: provides saved results
Reports and snapshots → Run result store: reads run data
Tracer plugin → Trace client: registers client
Trace client → Collector routes: sends traces
Collector routes → Trace ingestion: queues traces plugins/tracer/collector/routes-ingest.ts
Collector routes → Trajectory analysis: exports analysis plugins/tracer/collector/routes-ingest.ts
Query and analysis routes → Trajectory analysis: requests analysis plugins/tracer/collector/routes-query.ts
Collector routes → Trace store: records traces
Query and analysis routes → Trace store: queries traces
Trajectory viewer → Collector routes: views trajectories
AgentX performance benchmark → Agent and model runtimes: replays model requests README.md

Drawn by GitDiagram on November 10, 2026 from the repository's file tree, README and source files. Read as Markdown ·