# ydnyshhh/trace2eval architecture

> Architecture diagram of the GitHub repository ydnyshhh/trace2eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ydnyshhh/trace2eval
- Repository: https://github.com/ydnyshhh/trace2eval

## Overview

1. Trace2Eval is a local-first pipeline for converting real coding-agent runs into regression evals. Its architecture is organized as a sequence of artifact transformations: ingest raw agent logs, normalize them into a common timeline, detect likely failure hypotheses, generate compact evals, and replay those evals against later traces.

2. The system’s core boundary is the canonical trace model in `trace2eval/schemas.py`. Everything upstream is adapter-specific parsing, and everything downstream consumes these shared schemas. The project preserves unknown source fields in metadata so Codex/Claude inputs can be parsed defensively without losing information.

3. Ingestion is handled by adapter modules under `trace2eval/adapters/` and `trace2eval/capture/`. These cover Codex CLI rollout logs, Claude Code hook logs, Claude headless/programmatic outputs, and a generic trace format. This is the main external-input boundary: source-specific logs are converted into RawTrace JSON.

4. Normalization in `trace2eval/normalize.py` turns raw traces into a unified action/phase timeline. This is the point where heterogeneous agent events become comparable across tools and runs, enabling later mining and replay to operate on a single semantic shape rather than vendor-specific log formats.

5. Failure detection and hypothesis ranking are centralized in `trace2eval/detectors.py` and coordinated by `trace2eval/mining.py`. This layer scans normalized timelines for patterns like premature edits or missing verification, then produces candidate causal failure reports that explain where the run likely went wrong.

6. Eval synthesis lives in `trace2eval/generation.py`. It takes mined failure hypotheses and extracts a compact slice around the suspected onset to produce deterministic trajectory-level eval cases. The generated eval artifacts are designed to be small, inspectable regression tests rather than full reproduction logs.

7. Execution and comparison are split into two concerns: `trace2eval/runner.py` replays evals against trace sets, while `trace2eval/replay.py` and `trace2eval/counterfactual.py` provide human-facing diagnostics. Replay answers “does this eval fail on these traces?”, and counterfactual builds a symbolic corrected trajectory to check whether the hypothesis flips as expected.

8. Reporting and inspection are separate presentation layers. `trace2eval/report.py` renders terminal reports, and `trace2eval/inspect`-style behavior described in the README surfaces normalized timelines with detector markers and causal roles. This keeps analysis outputs distinct from the core pipeline artifacts.

9. Persistence is primarily file-based and local. The expected flow writes RawTrace, normalized traces, failure reports, generated eval YAML, and run results to `.trace2eval/` folders. Optional analytical indexing is provided by `trace2eval/storage/duckdb_store.py`, which adds a DuckDB-backed query layer without becoming the primary source of truth.

10. The CLI in `trace2eval/cli.py` is the orchestration boundary. It exposes the pipeline as discrete commands—init, doctor, ingest, normalize, mine, generate, run, replay, counterfactual, index, query, report—so operators can run the whole flow or inspect any stage independently.

11. The repository’s examples are part of the architecture, not just documentation. `examples/traces`, `examples/realistic`, `examples/fixtures`, and `examples/real_runs` provide synthetic, sanitized, and real-world traces that exercise the adapters and the pipeline end to end. Tests mirror this structure and validate each stage separately, which reinforces the intended stage boundaries.

12. In diagram terms, the main data flow is: external agent logs → adapter/capture parsing → RawTrace artifacts → normalized timelines → failure mining → eval generation → replay/reporting, with optional DuckDB indexing and counterfactual analysis branching off the normalized and generated artifacts.

Main entry points: [`trace2eval/cli.py`](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/cli.py) (CLI), [`trace2eval/adapters`](https://github.com/ydnyshhh/trace2eval/tree/main/trace2eval/adapters) (Adapters), [`trace2eval/schemas.py`](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/schemas.py) (Schemas), [`trace2eval/normalize.py`](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/normalize.py) (Normalize)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_cli["Orchestration"]
  node_node_cli["CLI<br/>commands<br/>[cli.py]"]
  node_node_doctor["Doctor<br/>environment check<br/>[doctor.py]"]
end

subgraph group_group_ingest["Ingest"]
  node_node_adapters["Adapters<br/>source parsers"]
  node_node_capture["Capture<br/>log collectors"]
  node_node_claude_adapter["Claude adapter<br/>parser<br/>[claude.py]"]
  node_node_codex_adapter["Codex adapter<br/>parser<br/>[codex.py]"]
  node_node_generic_adapter["Generic adapter<br/>parser<br/>[generic.py]"]
  node_node_claude_capture["Claude capture<br/>live logger<br/>[claude.py]"]
  node_node_codex_capture["Codex capture<br/>live logger<br/>[codex.py]"]
end

subgraph group_group_core["Core pipeline"]
  node_node_schemas["Schemas<br/>canonical model<br/>[schemas.py]"]
  node_node_normalize["Normalize<br/>timeline transform<br/>[normalize.py]"]
  node_node_detectors["Detectors<br/>failure heuristics<br/>[detectors.py]"]
  node_node_mining["Mining<br/>hypothesis ranking<br/>[mining.py]"]
  node_node_generation["Generate<br/>eval synthesis<br/>[generation.py]"]
end

subgraph group_group_exec["Replay & analysis"]
  node_node_runner["Runner<br/>eval execution<br/>[runner.py]"]
  node_node_replay["Replay<br/>comparison<br/>[replay.py]"]
  node_node_counterfactual["Counterfactual<br/>symbolic check<br/>[counterfactual.py]"]
  node_node_report["Report<br/>terminal output<br/>[report.py]"]
end

subgraph group_group_data["Data & examples"]
  node_node_duckdb[("DuckDB store<br/>query index<br/>[duckdb_store.py]")]
  node_node_io["IO<br/>artifact persistence<br/>[io.py]"]
  node_node_examples["Examples<br/>trace fixtures"]
end

node_node_cli -->|"doctor"| node_node_doctor
node_node_cli -->|"ingest"| node_node_adapters
node_node_cli -->|"normalize"| node_node_normalize
node_node_cli -->|"mine"| node_node_mining
node_node_cli -->|"generate"| node_node_generation
node_node_cli -->|"run"| node_node_runner
node_node_cli -->|"report"| node_node_report
node_node_adapters -->|"claude"| node_node_claude_adapter
node_node_adapters -->|"codex"| node_node_codex_adapter
node_node_adapters -->|"generic"| node_node_generic_adapter
node_node_capture -->|"claude"| node_node_claude_capture
node_node_capture -->|"codex"| node_node_codex_capture
node_node_claude_adapter -->|"raw trace"| node_node_schemas
node_node_codex_adapter -->|"raw trace"| node_node_schemas
node_node_generic_adapter -->|"raw trace"| node_node_schemas
node_node_claude_capture -->|"events"| node_node_claude_adapter
node_node_codex_capture -->|"rollout logs"| node_node_codex_adapter
node_node_schemas -->|"canonical input"| node_node_normalize
node_node_normalize -->|"timeline"| node_node_detectors
node_node_detectors -->|"signals"| node_node_mining
node_node_mining -->|"hypotheses"| node_node_generation
node_node_generation -->|"evals"| node_node_runner
node_node_runner -->|"results"| node_node_replay
node_node_mining -->|"hypothesis"| node_node_counterfactual
node_node_generation -->|"eval slice"| node_node_counterfactual
node_node_normalize -->|"index"| node_node_duckdb
node_node_schemas -->|"artifacts"| node_node_io
node_node_io -->|"fixtures"| node_node_examples
node_node_examples -->|"sample logs"| node_node_adapters
node_node_examples -->|"eval cases"| node_node_runner
node_node_report -->|"summaries"| node_node_io

click node_node_cli "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/cli.py"
click node_node_doctor "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/doctor.py"
click node_node_schemas "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/schemas.py"
click node_node_normalize "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/normalize.py"
click node_node_detectors "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/detectors.py"
click node_node_mining "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/mining.py"
click node_node_generation "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/generation.py"
click node_node_runner "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/runner.py"
click node_node_replay "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/replay.py"
click node_node_counterfactual "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/counterfactual.py"
click node_node_report "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/report.py"
click node_node_duckdb "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/storage/duckdb_store.py"
click node_node_io "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/io.py"
click node_node_adapters "https://github.com/ydnyshhh/trace2eval/tree/main/trace2eval/adapters"
click node_node_capture "https://github.com/ydnyshhh/trace2eval/tree/main/trace2eval/capture"
click node_node_claude_adapter "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/claude.py"
click node_node_codex_adapter "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/codex.py"
click node_node_generic_adapter "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/generic.py"
click node_node_claude_capture "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/capture/claude.py"
click node_node_codex_capture "https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/capture/codex.py"
click node_node_examples "https://github.com/ydnyshhh/trace2eval/tree/main/examples"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_cli,node_node_doctor toneBlue
class node_node_adapters,node_node_capture,node_node_claude_adapter,node_node_codex_adapter,node_node_generic_adapter,node_node_claude_capture,node_node_codex_capture toneAmber
class node_node_schemas,node_node_normalize,node_node_detectors,node_node_mining,node_node_generation toneMint
class node_node_runner,node_node_replay,node_node_counterfactual,node_node_report toneRose
class node_node_duckdb,node_node_io,node_node_examples toneIndigo
```

## Components

### Orchestration

Command-line entrypoints and operational commands.

- [CLI](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/cli.py): commands · Top-level command dispatcher for the pipeline. · `trace2eval/cli.py`
- [Doctor](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/doctor.py): environment check · Repository and runtime health checks. · `trace2eval/doctor.py`

### Ingest

Source-specific parsing and capture into raw artifacts.

- [Adapters](https://github.com/ydnyshhh/trace2eval/tree/main/trace2eval/adapters): source parsers · Adapter package for vendor and generic trace formats. · `trace2eval/adapters`
- [Capture](https://github.com/ydnyshhh/trace2eval/tree/main/trace2eval/capture): log collectors · Live capture helpers for Claude and Codex sources. · `trace2eval/capture`
- [Claude adapter](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/claude.py): parser · Parses Claude Code and headless trace formats. · `trace2eval/adapters/claude.py`
- [Codex adapter](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/codex.py): parser · Parses Codex CLI rollout logs into raw traces. · `trace2eval/adapters/codex.py`
- [Generic adapter](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/adapters/generic.py): parser · Fallback parser for generic trace input. · `trace2eval/adapters/generic.py`
- [Claude capture](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/capture/claude.py): live logger · Hooks and capture utilities for Claude runs. · `trace2eval/capture/claude.py`
- [Codex capture](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/capture/codex.py): live logger · Capture utilities for Codex rollout logs. · `trace2eval/capture/codex.py`

### Core pipeline

Canonical trace model and downstream transformations.

- [Schemas](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/schemas.py): canonical model · Shared trace and eval schemas that define the repository boundary. · `trace2eval/schemas.py`
- [Normalize](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/normalize.py): timeline transform · Converts raw traces into a unified action and phase timeline. · `trace2eval/normalize.py`
- [Detectors](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/detectors.py): failure heuristics · Finds likely failure patterns in normalized timelines. · `trace2eval/detectors.py`
- [Mining](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/mining.py): hypothesis ranking · Coordinates detector output into candidate failure reports. · `trace2eval/mining.py`
- [Generate](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/generation.py): eval synthesis · Builds compact deterministic eval cases from mined failures. · `trace2eval/generation.py`

### Replay & analysis

Execution, comparison, and human-facing diagnostics.

- [Runner](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/runner.py): eval execution · Replays generated evals against trace sets. · `trace2eval/runner.py`
- [Replay](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/replay.py): comparison · Checks eval behavior against concrete traces and outcomes. · `trace2eval/replay.py`
- [Counterfactual](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/counterfactual.py): symbolic check · Constructs a corrected trajectory to test whether a hypothesis flips. · `trace2eval/counterfactual.py`
- [Report](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/report.py): terminal output · Renders human-readable terminal reports. · `trace2eval/report.py`

### Data & examples

Persistent artifacts and bundled traces used to exercise the pipeline.

- [DuckDB store](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/storage/duckdb_store.py): query index · Optional analytical index for querying traces and runs. · `trace2eval/storage/duckdb_store.py`
- [IO](https://github.com/ydnyshhh/trace2eval/blob/main/trace2eval/io.py): artifact persistence · File-based read/write helpers for local trace artifacts. · `trace2eval/io.py`
- [Examples](https://github.com/ydnyshhh/trace2eval/tree/main/examples): trace fixtures · Synthetic, realistic, and real run examples that exercise the pipeline. · `examples`

## Connections

- CLI → Doctor: doctor · CLI exposes environment checks.
- CLI → Adapters: ingest · CLI dispatches source parsing commands.
- CLI → Normalize: normalize · CLI can run normalization directly.
- CLI → Mining: mine · CLI can run failure detection and ranking.
- CLI → Generate: generate · CLI can synthesize eval artifacts.
- CLI → Runner: run · CLI replays evals over traces.
- CLI → Report: report · CLI renders inspection and summary output.
- Adapters → Claude adapter: claude · Claude-specific parsers live in the adapters package.
- Adapters → Codex adapter: codex · Codex-specific parsers live in the adapters package.
- Adapters → Generic adapter: generic · Generic parser handles non-vendor trace input.
- Capture → Claude capture: claude · Claude capture helpers collect live logs.
- Capture → Codex capture: codex · Codex capture helpers collect live logs.
- Claude adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Codex adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Generic adapter → Schemas: raw trace · Adapter output is normalized into the shared schema boundary.
- Claude capture → Claude adapter: events · Captured Claude logs feed Claude parsing.
- Codex capture → Codex adapter: rollout logs · Captured Codex logs feed Codex parsing.
- Schemas → Normalize: canonical input · Normalization consumes the canonical raw trace model.
- Normalize → Detectors: timeline · Detectors analyze the unified action and phase timeline.
- Detectors → Mining: signals · Detector findings are ranked into candidate failure hypotheses.
- Mining → Generate: hypotheses · Mining output is sliced into compact eval cases.
- Generate → Runner: evals · Generated eval artifacts are executed by the runner.
- Runner → Replay: results · Runner outcomes are inspected through replay logic.
- Mining → Counterfactual: hypothesis · Counterfactual analysis branches from mined failure reports.
- Generate → Counterfactual: eval slice · Counterfactual checks the synthesized trajectory around the failure onset.
- Normalize → DuckDB store: index · Normalized traces can be loaded into an optional DuckDB store.
- Schemas → IO: artifacts · Shared schemas are persisted through local file helpers.
- IO → Examples: fixtures · Examples and runs provide local artifact inputs and outputs.
- Examples → Adapters: sample logs · Bundled examples exercise source-specific parsing.
- Examples → Runner: eval cases · Example evals and runs validate replay behavior.
- Report → IO: summaries · Reports summarize artifacts stored on disk.
