How minghuichen43/tep works
1. What this repository is
TEP is an experiment framework for “textual equilibrium propagation” over compound AI workflows. The core architectural idea is a two-phase optimization loop: a free phase where prompts/workflow state settle locally, and a nudged phase where task feedback updates bounded parts of the system. The repo is organized around running that loop on four benchmark tasks rather than around a single application backend.
2. Top-level flow
The main runtime path is: CLI entrypoint → config loading → task/workflow construction → prompt/state optimization loop → metric evaluation → result output. Most of the repo’s code sits in src/tep/, with datasets and benchmark-specific logic separated under src/tep/tasks/, and shared optimization/runtime machinery under src/tep/runtime/.
3. Command-line orchestration
src/tep/cli/ contains the primary user-facing entrypoints:
- train.py for running TEP optimization
- evaluate.py for scoring a configured workflow
- run_main_results.py for multi-benchmark experiment execution
- build_split_manifest.py for producing or managing dataset split manifests
These scripts are the boundary between experiment configuration and the actual optimization/evaluation pipeline.
4. Shared runtime layer
src/tep/runtime/ is the common orchestration layer. It contains:
- core.py: the central execution/optimization loop
- prompt_state.py: representation and mutation of prompt state during free/nudged phases
- registry.py: registration/lookup for workflows, datasets, or task components
- split_utils.py: dataset split handling and manifest logic
This layer is where benchmark-specific workflows are normalized into a common execution model.
5. Benchmark/task subsystem
Each benchmark has its own package under src/tep/tasks/ with a similar shape: dataset loading, workflow composition, and sometimes task-specific evaluation helpers.
- hotpotqa/
- pubmed/
- stark/
- bigcodebench/
The repeated structure suggests that each task is an isolated boundary around its data source and workflow definition, while still using the same runtime and metrics infrastructure.
6. Task-specific architecture differences
The tasks differ materially in how they obtain data and what the workflow is optimizing:
- HotpotQA: multi-hop QA, loaded from HuggingFace datasets
- PubMedQA: biomedical QA, using local JSONL files under data/pubmed/
- STARK-PRIME: retrieval-style QA, depends on stark_qa and possibly external embedding assets for VSS mode
- BigCodeBench: code generation/repair, loaded from HuggingFace datasets with a fixed benchmark split
This means the workflow boundary is stable, but the dataset/retrieval boundary changes by benchmark.
7. Evaluation and scoring
src/tep/metrics/ holds reusable scoring functions:
- f1.py
- mrr.py
- pass_rate.py
These map to the different task types: QA-style overlap scoring, retrieval ranking quality, and code-task pass/fail rates. The metric layer is separate from workflow execution so the same runtime can be reused across heterogeneous benchmarks.
8. Data and split management
The repository includes local benchmark assets and data:
- data/pubmed/ contains PubMedQA train/test JSONL files
- src/tep/tasks/bigcodebench/assets/ contains prompt initialization and a fixed split manifest
- configs/*.yaml define task-specific experiments and likely encode dataset paths, model settings, and optimization parameters
The presence of split utilities and manifests indicates reproducible benchmark partitioning is treated as a first-class part of the architecture.
9. External runtime and model access
The README indicates model access is configured through environment variables, typically via .env, with OpenRouter as the default LLM gateway. The repo also vendors a TextGrad runtime under src/tep/vendor/, which acts as an internal dependency layer for prompt optimization primitives. This means the system is designed to run locally while depending on remote LLM inference as an external service.
10. Testing and repo boundaries
Tests are minimal but focused on behavior at the workflow/runtime boundary:
- tests/test_evaluate_prompt_state.py
- tests/test_stark_workflow.py
That suggests the most important correctness points are prompt-state evaluation and task workflow wiring, not isolated algorithmic units. For an architecture diagram, the main boundaries to show are: CLI, shared runtime, task-specific workflows/datasets, metrics, config/data assets, and external model/data services.
Main entry points: core.py Core loop · registry.py Registry · configs Configs · workflow.py Hotpot flow
Components 28
CLI
User-facing experiment commands
- Train · Runs TEP optimization
src/tep/cli/train.py - Evaluate · Scores a configured workflow
src/tep/cli/evaluate.py - Main results · Runs multi-benchmark experiments
src/tep/cli/run_main_results.py - Split manifest · Builds dataset split manifests
src/tep/cli/build_split_manifest.py
Runtime
Shared optimization and orchestration
- Core loop · Free-phase and nudged-phase execution loop
src/tep/runtime/core.py - Prompt state · Mutable prompt/workflow state across phases
src/tep/runtime/prompt_state.py - Registry · Lookup and registration boundary
src/tep/runtime/registry.py - Split utils · Dataset split and manifest handling
src/tep/runtime/split_utils.py
Tasks
Benchmark-specific datasets and workflows
- Hotpot data · HotpotQA data loading
src/tep/tasks/hotpotqa/dataset.py - Hotpot flow · Multi-hop QA workflow composition
src/tep/tasks/hotpotqa/workflow.py - PubMed data · Local PubMedQA JSONL loading
src/tep/tasks/pubmed/dataset.py - PubMed flow · Biomedical QA workflow composition
src/tep/tasks/pubmed/workflow.py - PubMed eval · Task-specific PubMed evaluation helpers
src/tep/tasks/pubmed/eval.py - STARK data · STARK-PRIME retrieval data loading
src/tep/tasks/stark/dataset.py - STARK flow · Retrieval-style QA workflow composition
src/tep/tasks/stark/workflow.py - BigCode data · BigCodeBench benchmark loading
src/tep/tasks/bigcodebench/dataset.py - BigCode flow · Code generation and repair workflow
src/tep/tasks/bigcodebench/workflow.py - BigCode eval · Task-specific code evaluation helpers
src/tep/tasks/bigcodebench/eval.py
Metrics
Reusable scoring functions
Assets
Configs, local data, and vendored runtime
- Configs · Task-specific experiment YAMLs
configs - PubMed files · PubMedQA train/test JSONL files
data/pubmed - BigCode assets · Prompt init and fixed split manifest
src/tep/tasks/bigcodebench/assets - TextGrad · Internal prompt-optimization primitives
src/tep/vendor - LLM gateway · Remote model access via environment configuration
- Dataset hubs · HuggingFace and other external data sources
- stark_qa · STARK benchmark dependency
Connections 32
- Train → Core loop: starts · Launches the optimization loop
- Evaluate → Core loop: scores · Runs configured evaluation through the shared runtime
- Main results → Core loop: orchestrates · Drives repeated benchmark executions
- Split manifest → Split utils: builds · Produces or manages split manifests
- Core loop → Prompt state: updates · Advances prompt state across free and nudged phases
- Core loop → Registry: resolves · Finds workflows and task components
- Core loop → LLM gateway: calls · Uses remote model inference
- Core loop → TextGrad: uses · Relies on vendored prompt optimization primitives
- Registry → Hotpot flow: loads · Registers benchmark workflow
- Registry → PubMed flow: loads · Registers benchmark workflow
- Registry → STARK flow: loads · Registers benchmark workflow
- Registry → BigCode flow: loads · Registers benchmark workflow
- Hotpot flow → Hotpot data: reads · Consumes HotpotQA examples
- Hotpot data → Dataset hubs: sources · Loads benchmark data from external datasets
- PubMed flow → PubMed data: reads · Consumes local PubMedQA data
- PubMed data → PubMed files: loads · Reads JSONL files from the repository
- PubMed flow → PubMed eval: delegates · Uses task-specific PubMed evaluation helpers
- PubMed eval → F1: scores · Evaluates QA overlap
- STARK flow → STARK data: reads · Consumes STARK-PRIME data
- STARK data → stark_qa: depends on · Uses the external STARK package
- STARK flow → MRR: scores · Uses ranking metric for retrieval-style evaluation
- BigCode flow → BigCode data: reads · Consumes BigCodeBench data
- BigCode data → Dataset hubs: sources · Loads benchmark data from external datasets
- BigCode flow → BigCode eval: delegates · Uses task-specific code evaluation helpers
- BigCode eval → Pass rate: scores · Evaluates pass/fail outcomes
- Hotpot flow → F1: scores · Uses QA overlap scoring
- Configs → Train: configures · Task experiment settings
- Configs → Evaluate: configures · Task experiment settings
- Configs → Main results: configures · Benchmark run settings
- Configs → Split manifest: configures · Split-manifest settings
- BigCode assets → BigCode data: initializes · Provides prompt init and split manifest
- BigCode assets → Split utils: feeds · Supplies reproducible split metadata