How ydnyshhh/trace2eval works
1. Trace2Eval is a local-first pipeline for converting real coding-agent runs into regression evals. Its architecture is organized as a sequence of artifact transformations: ingest raw agent logs, normalize them into a common timeline, detect likely failure hypotheses, generate compact evals, and replay those evals against later traces.
2. The system’s core boundary is the canonical trace model in trace2eval/schemas.py. Everything upstream is adapter-specific parsing, and everything downstream consumes these shared schemas. The project preserves unknown source fields in metadata so Codex/Claude inputs can be parsed defensively without losing information.
3. Ingestion is handled by adapter modules under trace2eval/adapters/ and trace2eval/capture/. These cover Codex CLI rollout logs, Claude Code hook logs, Claude headless/programmatic outputs, and a generic trace format. This is the main external-input boundary: source-specific logs are converted into RawTrace JSON.
4. Normalization in trace2eval/normalize.py turns raw traces into a unified action/phase timeline. This is the point where heterogeneous agent events become comparable across tools and runs, enabling later mining and replay to operate on a single semantic shape rather than vendor-specific log formats.
5. Failure detection and hypothesis ranking are centralized in trace2eval/detectors.py and coordinated by trace2eval/mining.py. This layer scans normalized timelines for patterns like premature edits or missing verification, then produces candidate causal failure reports that explain where the run likely went wrong.
6. Eval synthesis lives in trace2eval/generation.py. It takes mined failure hypotheses and extracts a compact slice around the suspected onset to produce deterministic trajectory-level eval cases. The generated eval artifacts are designed to be small, inspectable regression tests rather than full reproduction logs.
7. Execution and comparison are split into two concerns: trace2eval/runner.py replays evals against trace sets, while trace2eval/replay.py and trace2eval/counterfactual.py provide human-facing diagnostics. Replay answers “does this eval fail on these traces?”, and counterfactual builds a symbolic corrected trajectory to check whether the hypothesis flips as expected.
8. Reporting and inspection are separate presentation layers. trace2eval/report.py renders terminal reports, and trace2eval/inspect-style behavior described in the README surfaces normalized timelines with detector markers and causal roles. This keeps analysis outputs distinct from the core pipeline artifacts.
9. Persistence is primarily file-based and local. The expected flow writes RawTrace, normalized traces, failure reports, generated eval YAML, and run results to .trace2eval/ folders. Optional analytical indexing is provided by trace2eval/storage/duckdb_store.py, which adds a DuckDB-backed query layer without becoming the primary source of truth.
10. The CLI in trace2eval/cli.py is the orchestration boundary. It exposes the pipeline as discrete commands—init, doctor, ingest, normalize, mine, generate, run, replay, counterfactual, index, query, report—so operators can run the whole flow or inspect any stage independently.
11. The repository’s examples are part of the architecture, not just documentation. examples/traces, examples/realistic, examples/fixtures, and examples/real_runs provide synthetic, sanitized, and real-world traces that exercise the adapters and the pipeline end to end. Tests mirror this structure and validate each stage separately, which reinforces the intended stage boundaries.
12. In diagram terms, the main data flow is: external agent logs → adapter/capture parsing → RawTrace artifacts → normalized timelines → failure mining → eval generation → replay/reporting, with optional DuckDB indexing and counterfactual analysis branching off the normalized and generated artifacts.
Main entry points: cli.py CLI · adapters Adapters · schemas.py Schemas · normalize.py Normalize
Components 21
Orchestration
Command-line entrypoints and operational commands.
Ingest
Source-specific parsing and capture into raw artifacts.
- Adapters · Adapter package for vendor and generic trace formats.
trace2eval/adapters - Capture · Live capture helpers for Claude and Codex sources.
trace2eval/capture - Claude adapter · Parses Claude Code and headless trace formats.
trace2eval/adapters/claude.py - Codex adapter · Parses Codex CLI rollout logs into raw traces.
trace2eval/adapters/codex.py - Generic adapter · Fallback parser for generic trace input.
trace2eval/adapters/generic.py - Claude capture · Hooks and capture utilities for Claude runs.
trace2eval/capture/claude.py - Codex capture · Capture utilities for Codex rollout logs.
trace2eval/capture/codex.py
Core pipeline
Canonical trace model and downstream transformations.
- Schemas · Shared trace and eval schemas that define the repository boundary.
trace2eval/schemas.py - Normalize · Converts raw traces into a unified action and phase timeline.
trace2eval/normalize.py - Detectors · Finds likely failure patterns in normalized timelines.
trace2eval/detectors.py - Mining · Coordinates detector output into candidate failure reports.
trace2eval/mining.py - Generate · Builds compact deterministic eval cases from mined failures.
trace2eval/generation.py
Replay & analysis
Execution, comparison, and human-facing diagnostics.
- Runner · Replays generated evals against trace sets.
trace2eval/runner.py - Replay · Checks eval behavior against concrete traces and outcomes.
trace2eval/replay.py - Counterfactual · Constructs a corrected trajectory to test whether a hypothesis flips.
trace2eval/counterfactual.py - Report · Renders human-readable terminal reports.
trace2eval/report.py
Data & examples
Persistent artifacts and bundled traces used to exercise the pipeline.
- DuckDB store · Optional analytical index for querying traces and runs.
trace2eval/storage/duckdb_store.py - IO · File-based read/write helpers for local trace artifacts.
trace2eval/io.py - Examples · Synthetic, realistic, and real run examples that exercise the pipeline.
examples
Connections 31
- CLI → Doctor: doctor · CLI exposes environment checks.
- CLI → Adapters: ingest · CLI dispatches source parsing commands.
- CLI → Normalize: normalize · CLI can run normalization directly.
- CLI → Mining: mine · CLI can run failure detection and ranking.
- CLI → Generate: generate · CLI can synthesize eval artifacts.
- CLI → Runner: run · CLI replays evals over traces.
- CLI → Report: report · CLI renders inspection and summary output.
- Adapters → Claude adapter: claude · Claude-specific parsers live in the adapters package.
- Adapters → Codex adapter: codex · Codex-specific parsers live in the adapters package.
- Adapters → Generic adapter: generic · Generic parser handles non-vendor trace input.
- Capture → Claude capture: claude · Claude capture helpers collect live logs.
- Capture → Codex capture: codex · Codex capture helpers collect live logs.
- Claude adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Codex adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Generic adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Claude capture → Claude adapter: events · Captured Claude logs feed Claude parsing.
- Codex capture → Codex adapter: rollout logs · Captured Codex logs feed Codex parsing.
- Schemas → Normalize: canonical input · Normalization consumes the canonical raw trace model.
- Normalize → Detectors: timeline · Detectors analyze the unified action and phase timeline.
- Detectors → Mining: signals · Detector findings are ranked into candidate failure hypotheses.
- Mining → Generate: hypotheses · Mining output is sliced into compact eval cases.
- Generate → Runner: evals · Generated eval artifacts are executed by the runner.
- Runner → Replay: results · Runner outcomes are inspected through replay logic.
- Mining → Counterfactual: hypothesis · Counterfactual analysis branches from mined failure reports.
- Generate → Counterfactual: eval slice · Counterfactual checks the synthesized trajectory around the failure onset.
- Normalize → DuckDB store: index · Normalized traces can be loaded into an optional DuckDB store.
- Schemas → IO: artifacts · Shared schemas are persisted through local file helpers.
- IO → Examples: fixtures · Examples and runs provide local artifact inputs and outputs.
- Examples → Adapters: sample logs · Bundled examples exercise source-specific parsing.
- Examples → Runner: eval cases · Example evals and runs validate replay behavior.
- Report → IO: summaries · Reports summarize artifacts stored on disk.