How jerryyan123/traceml works
TraceML Toolkit turns an ML-agent run into a versioned trajectory, labels its code states and edits, and produces dataset-schema tables and a behavior report compared with human cohorts. The main workflow is orchestrated by pipeline.py; record.py is an optional way to capture and grade a run before analysis. The graph preserves README-documented input layouts and report capabilities. Some subsystem wiring is based on the README because its implementation was not sampled.
Main entry points: runner.py Label runner · pipeline.py Analysis pipeline · report.py Behavior report · cli.py Command interface
Components 25
Run capture
- Run recorder
traceml_toolkit/record.py - Command interface
traceml_toolkit/cli.py
Trajectory extraction
- Analysis pipeline
traceml_toolkit/pipeline.py - Layout detector
traceml_toolkit/detect.py - Trajectory extractor
traceml_toolkit/extract.py - Trajectory and versions
- Released-run importer
traceml_toolkit/from_released.py
Labeling
- Label runner
traceml_toolkit/labeling/runner.py - State and edit inputs
traceml_toolkit/labeling/inputs.py - Prompt rendering
traceml_toolkit/labeling/render.py - Label prompts
traceml_toolkit/labeling/prompts - Backend selection
traceml_toolkit/labeling/backends - vLLM backend
traceml_toolkit/labeling/backends/vllm_backend.py - Transformers backend
traceml_toolkit/labeling/backends/hf_backend.py - Label files
Comparison and reporting
- Cohort comparisons
traceml_toolkit/cohorts.py - Behavior report
traceml_toolkit/report.py - Reports and figures
Data and model support
- Dataset schema
traceml_toolkit/schema.py - Parquet tables
- Models and assets
traceml_toolkit/assets.py
Other components
- CLI user
- Agent run
- MLE-bench
- Hugging Face
Connections 33
- CLI user → Command interface: invokes
- CLI user → Run recorder: starts capture
- Run recorder → Agent run: runs command
- Run recorder → MLE-bench: grades submissions
- Command interface → Analysis pipeline: runs analysis
- Command interface → Run recorder: starts recording
- Analysis pipeline → Layout detector: detects layout
- Analysis pipeline → Trajectory extractor: dispatches extraction
- Trajectory extractor → Agent run: reads run
- Trajectory extractor → MLE-bench: regrades submissions
- Trajectory extractor → Trajectory and versions: writes versions
- Analysis pipeline → Label runner: runs labeling
- Label runner → Trajectory and versions: reads trajectory
- Label runner → State and edit inputs: builds records
- Label runner → Prompt rendering: renders prompts
- Prompt rendering → Label prompts: uses templates
- Label runner → Backend selection: selects backend
- Backend selection → vLLM backend: dispatches
- Backend selection → Transformers backend: dispatches
- Label runner → Models and assets: resolves models
- Models and assets → Hugging Face: downloads assets
- Label runner → Label files: writes labels
- Analysis pipeline → Dataset schema: converts outputs
- Dataset schema → Parquet tables: writes tables
- Analysis pipeline → Behavior report: generates report
- Behavior report → Trajectory and versions: reads trajectory
- Behavior report → Label files: reads labels
- Behavior report → Cohort comparisons: loads comparisons
- Cohort comparisons → Hugging Face: loads references
- Behavior report → Reports and figures: writes results
- Command interface → Released-run importer: imports released run
- Released-run importer → Hugging Face: loads released data
- Released-run importer → Trajectory and versions: writes trajectory