# jerryyan123/traceml architecture

> Architecture diagram of the GitHub repository jerryyan123/traceml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jerryyan123/traceml
- Repository: https://github.com/jerryyan123/traceml

## Overview

TraceML Toolkit turns an ML-agent run into a versioned trajectory, labels its code states and edits, and produces dataset-schema tables and a behavior report compared with human cohorts. The main workflow is orchestrated by `pipeline.py`; `record.py` is an optional way to capture and grade a run before analysis. The graph preserves README-documented input layouts and report capabilities. Some subsystem wiring is based on the README because its implementation was not sampled.

Main entry points: [`traceml_toolkit/labeling/runner.py`](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/runner.py) (Label runner), [`traceml_toolkit/pipeline.py`](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/pipeline.py) (Analysis pipeline), [`traceml_toolkit/report.py`](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/report.py) (Behavior report), [`traceml_toolkit/cli.py`](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/cli.py) (Command interface)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Run capture"]
  node_recorder["Run recorder<br/>[record.py]"]
  node_cli["Command interface<br/>[cli.py]"]
end

subgraph group_trajectory["Trajectory extraction"]
  node_pipeline["Analysis pipeline<br/>[pipeline.py]"]
  node_detector["Layout detector<br/>[detect.py]"]
  node_extractor["Trajectory extractor<br/>[extract.py]"]
  node_trajectory_data["Trajectory and versions"]
  node_released["Released-run importer<br/>[from_released.py]"]
end

subgraph group_labeling["Labeling"]
  node_label_runner["Label runner<br/>[runner.py]"]
  node_label_inputs["State and edit inputs<br/>[inputs.py]"]
  node_prompt_render["Prompt rendering<br/>[render.py]"]
  node_label_prompts["Label prompts"]
  node_backend["Backend selection"]
  node_vllm["vLLM backend<br/>[vllm_backend.py]"]
  node_transformers["Transformers backend<br/>[hf_backend.py]"]
  node_label_data["Label files"]
end

subgraph group_analysis["Comparison and reporting"]
  node_cohorts["Cohort comparisons<br/>[cohorts.py]"]
  node_report["Behavior report<br/>[report.py]"]
  node_report_files["Reports and figures"]
end

subgraph group_support["Data and model support"]
  node_schema["Dataset schema<br/>[schema.py]"]
  node_tables["Parquet tables"]
  node_assets["Models and assets<br/>[assets.py]"]
end

node_operator(("CLI user"))
node_agent["Agent run"]
node_mlebench{{"MLE-bench"}}
node_huggingface{{"Hugging Face"}}

node_operator -->|"invokes"| node_cli
node_operator -->|"starts capture"| node_recorder
node_recorder -->|"runs command"| node_agent
node_recorder -.->|"grades submissions"| node_mlebench
node_cli -->|"runs analysis"| node_pipeline
node_cli -->|"starts recording"| node_recorder
node_pipeline -->|"detects layout"| node_detector
node_pipeline -->|"dispatches extraction"| node_extractor
node_extractor -->|"reads run"| node_agent
node_extractor -.->|"regrades submissions"| node_mlebench
node_extractor -->|"writes versions"| node_trajectory_data
node_pipeline -->|"runs labeling"| node_label_runner
node_label_runner -->|"reads trajectory"| node_trajectory_data
node_label_runner -->|"builds records"| node_label_inputs
node_label_runner -->|"renders prompts"| node_prompt_render
node_prompt_render -->|"uses templates"| node_label_prompts
node_label_runner -->|"selects backend"| node_backend
node_backend -->|"dispatches"| node_vllm
node_backend -->|"dispatches"| node_transformers
node_label_runner -->|"resolves models"| node_assets
node_assets -->|"downloads assets"| node_huggingface
node_label_runner -->|"writes labels"| node_label_data
node_pipeline -->|"converts outputs"| node_schema
node_schema -->|"writes tables"| node_tables
node_pipeline -->|"generates report"| node_report
node_report -->|"reads trajectory"| node_trajectory_data
node_report -->|"reads labels"| node_label_data
node_report -->|"loads comparisons"| node_cohorts
node_cohorts -->|"loads references"| node_huggingface
node_report -->|"writes results"| node_report_files
node_cli -->|"imports released run"| node_released
node_released -->|"loads released data"| node_huggingface
node_released -->|"writes trajectory"| node_trajectory_data

click node_recorder "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/record.py"
click node_cli "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/cli.py"
click node_pipeline "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/pipeline.py"
click node_detector "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/detect.py"
click node_extractor "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/extract.py"
click node_label_runner "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/runner.py"
click node_label_inputs "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/inputs.py"
click node_prompt_render "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/render.py"
click node_label_prompts "https://github.com/jerryyan123/traceml/tree/main/traceml_toolkit/labeling/prompts"
click node_backend "https://github.com/jerryyan123/traceml/tree/main/traceml_toolkit/labeling/backends"
click node_vllm "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/backends/vllm_backend.py"
click node_transformers "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/backends/hf_backend.py"
click node_schema "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/schema.py"
click node_assets "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/assets.py"
click node_cohorts "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/cohorts.py"
click node_report "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/report.py"
click node_released "https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/from_released.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_recorder,node_cli,node_operator toneBlue
class node_pipeline,node_detector,node_extractor,node_trajectory_data,node_released toneAmber
class node_label_runner,node_label_inputs,node_prompt_render,node_label_prompts,node_backend,node_vllm,node_transformers,node_label_data toneMint
class node_cohorts,node_report,node_report_files toneRose
class node_schema,node_tables,node_assets,node_agent,node_mlebench,node_huggingface toneIndigo
```

## Components

### Run capture

- [Run recorder](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/record.py): `traceml_toolkit/record.py`
- [Command interface](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/cli.py): `traceml_toolkit/cli.py`

### Trajectory extraction

- [Analysis pipeline](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/pipeline.py): `traceml_toolkit/pipeline.py`
- [Layout detector](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/detect.py): `traceml_toolkit/detect.py`
- [Trajectory extractor](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/extract.py): `traceml_toolkit/extract.py`
- **Trajectory and versions**
- [Released-run importer](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/from_released.py): `traceml_toolkit/from_released.py`

### Labeling

- [Label runner](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/runner.py): `traceml_toolkit/labeling/runner.py`
- [State and edit inputs](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/inputs.py): `traceml_toolkit/labeling/inputs.py`
- [Prompt rendering](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/render.py): `traceml_toolkit/labeling/render.py`
- [Label prompts](https://github.com/jerryyan123/traceml/tree/main/traceml_toolkit/labeling/prompts): `traceml_toolkit/labeling/prompts`
- [Backend selection](https://github.com/jerryyan123/traceml/tree/main/traceml_toolkit/labeling/backends): `traceml_toolkit/labeling/backends`
- [vLLM backend](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/backends/vllm_backend.py): `traceml_toolkit/labeling/backends/vllm_backend.py`
- [Transformers backend](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/labeling/backends/hf_backend.py): `traceml_toolkit/labeling/backends/hf_backend.py`
- **Label files**

### Comparison and reporting

- [Cohort comparisons](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/cohorts.py): `traceml_toolkit/cohorts.py`
- [Behavior report](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/report.py): `traceml_toolkit/report.py`
- **Reports and figures**

### Data and model support

- [Dataset schema](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/schema.py): `traceml_toolkit/schema.py`
- **Parquet tables**
- [Models and assets](https://github.com/jerryyan123/traceml/blob/main/traceml_toolkit/assets.py): `traceml_toolkit/assets.py`

### Other components

- **CLI user**
- **Agent run**
- **MLE-bench**
- **Hugging Face**

## Connections

- CLI user → Command interface: invokes
- CLI user → Run recorder: starts capture
- Run recorder → Agent run: runs command
- Run recorder → MLE-bench: grades submissions
- Command interface → Analysis pipeline: runs analysis
- Command interface → Run recorder: starts recording
- Analysis pipeline → Layout detector: detects layout
- Analysis pipeline → Trajectory extractor: dispatches extraction
- Trajectory extractor → Agent run: reads run
- Trajectory extractor → MLE-bench: regrades submissions
- Trajectory extractor → Trajectory and versions: writes versions
- Analysis pipeline → Label runner: runs labeling
- Label runner → Trajectory and versions: reads trajectory
- Label runner → State and edit inputs: builds records
- Label runner → Prompt rendering: renders prompts
- Prompt rendering → Label prompts: uses templates
- Label runner → Backend selection: selects backend
- Backend selection → vLLM backend: dispatches
- Backend selection → Transformers backend: dispatches
- Label runner → Models and assets: resolves models
- Models and assets → Hugging Face: downloads assets
- Label runner → Label files: writes labels
- Analysis pipeline → Dataset schema: converts outputs
- Dataset schema → Parquet tables: writes tables
- Analysis pipeline → Behavior report: generates report
- Behavior report → Trajectory and versions: reads trajectory
- Behavior report → Label files: reads labels
- Behavior report → Cohort comparisons: loads comparisons
- Cohort comparisons → Hugging Face: loads references
- Behavior report → Reports and figures: writes results
- Command interface → Released-run importer: imports released run
- Released-run importer → Hugging Face: loads released data
- Released-run importer → Trajectory and versions: writes trajectory