# minghuichen43/tep architecture

> Architecture diagram of the GitHub repository minghuichen43/tep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minghuichen43/tep
- Repository: https://github.com/minghuichen43/tep

## Overview

1. **What this repository is**
   TEP is an experiment framework for “textual equilibrium propagation” over compound AI workflows. The core architectural idea is a two-phase optimization loop: a **free phase** where prompts/workflow state settle locally, and a **nudged phase** where task feedback updates bounded parts of the system. The repo is organized around running that loop on four benchmark tasks rather than around a single application backend.

2. **Top-level flow**
   The main runtime path is: **CLI entrypoint → config loading → task/workflow construction → prompt/state optimization loop → metric evaluation → result output**. Most of the repo’s code sits in `src/tep/`, with datasets and benchmark-specific logic separated under `src/tep/tasks/`, and shared optimization/runtime machinery under `src/tep/runtime/`.

3. **Command-line orchestration**
   `src/tep/cli/` contains the primary user-facing entrypoints:
   - `train.py` for running TEP optimization
   - `evaluate.py` for scoring a configured workflow
   - `run_main_results.py` for multi-benchmark experiment execution
   - `build_split_manifest.py` for producing or managing dataset split manifests  
   These scripts are the boundary between experiment configuration and the actual optimization/evaluation pipeline.

4. **Shared runtime layer**
   `src/tep/runtime/` is the common orchestration layer. It contains:
   - `core.py`: the central execution/optimization loop
   - `prompt_state.py`: representation and mutation of prompt state during free/nudged phases
   - `registry.py`: registration/lookup for workflows, datasets, or task components
   - `split_utils.py`: dataset split handling and manifest logic  
   This layer is where benchmark-specific workflows are normalized into a common execution model.

5. **Benchmark/task subsystem**
   Each benchmark has its own package under `src/tep/tasks/` with a similar shape: dataset loading, workflow composition, and sometimes task-specific evaluation helpers.
   - `hotpotqa/`
   - `pubmed/`
   - `stark/`
   - `bigcodebench/`  
   The repeated structure suggests that each task is an isolated boundary around its data source and workflow definition, while still using the same runtime and metrics infrastructure.

6. **Task-specific architecture differences**
   The tasks differ materially in how they obtain data and what the workflow is optimizing:
   - **HotpotQA**: multi-hop QA, loaded from HuggingFace `datasets`
   - **PubMedQA**: biomedical QA, using local JSONL files under `data/pubmed/`
   - **STARK-PRIME**: retrieval-style QA, depends on `stark_qa` and possibly external embedding assets for VSS mode
   - **BigCodeBench**: code generation/repair, loaded from HuggingFace `datasets` with a fixed benchmark split  
   This means the workflow boundary is stable, but the dataset/retrieval boundary changes by benchmark.

7. **Evaluation and scoring**
   `src/tep/metrics/` holds reusable scoring functions:
   - `f1.py`
   - `mrr.py`
   - `pass_rate.py`  
   These map to the different task types: QA-style overlap scoring, retrieval ranking quality, and code-task pass/fail rates. The metric layer is separate from workflow execution so the same runtime can be reused across heterogeneous benchmarks.

8. **Data and split management**
   The repository includes local benchmark assets and data:
   - `data/pubmed/` contains PubMedQA train/test JSONL files
   - `src/tep/tasks/bigcodebench/assets/` contains prompt initialization and a fixed split manifest
   - `configs/*.yaml` define task-specific experiments and likely encode dataset paths, model settings, and optimization parameters  
   The presence of split utilities and manifests indicates reproducible benchmark partitioning is treated as a first-class part of the architecture.

9. **External runtime and model access**
   The README indicates model access is configured through environment variables, typically via `.env`, with OpenRouter as the default LLM gateway. The repo also vendors a TextGrad runtime under `src/tep/vendor/`, which acts as an internal dependency layer for prompt optimization primitives. This means the system is designed to run locally while depending on remote LLM inference as an external service.

10. **Testing and repo boundaries**
   Tests are minimal but focused on behavior at the workflow/runtime boundary:
   - `tests/test_evaluate_prompt_state.py`
   - `tests/test_stark_workflow.py`  
   That suggests the most important correctness points are prompt-state evaluation and task workflow wiring, not isolated algorithmic units. For an architecture diagram, the main boundaries to show are: **CLI**, **shared runtime**, **task-specific workflows/datasets**, **metrics**, **config/data assets**, and **external model/data services**.

Main entry points: [`src/tep/runtime/core.py`](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/core.py) (Core loop), [`src/tep/runtime/registry.py`](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/registry.py) (Registry), [`configs`](https://github.com/minghuichen43/tep/tree/main/configs) (Configs), [`src/tep/tasks/hotpotqa/workflow.py`](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/hotpotqa/workflow.py) (Hotpot flow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI"]
  node_cli_train["Train<br/>cli<br/>[train.py]"]
  node_cli_evaluate["Evaluate<br/>cli<br/>[evaluate.py]"]
  node_cli_results["Main results<br/>cli"]
  node_cli_manifest["Split manifest<br/>cli"]
end

subgraph group_runtime["Runtime"]
  node_runtime_core(("Core loop<br/>runtime<br/>[core.py]"))
  node_prompt_state["Prompt state<br/>runtime state<br/>[prompt_state.py]"]
  node_registry[("Registry<br/>[registry.py]")]
  node_split_utils["Split utils<br/>runtime splits<br/>[split_utils.py]"]
end

subgraph group_tasks["Tasks"]
  node_hotpot_dataset["Hotpot data<br/>dataset<br/>[dataset.py]"]
  node_hotpot_workflow["Hotpot flow<br/>workflow<br/>[workflow.py]"]
  node_pubmed_dataset["PubMed data<br/>dataset<br/>[dataset.py]"]
  node_pubmed_workflow["PubMed flow<br/>workflow<br/>[workflow.py]"]
  node_pubmed_eval["PubMed eval<br/>task eval<br/>[eval.py]"]
  node_stark_dataset["STARK data<br/>dataset<br/>[dataset.py]"]
  node_stark_workflow["STARK flow<br/>workflow<br/>[workflow.py]"]
  node_bigcode_dataset["BigCode data<br/>dataset<br/>[dataset.py]"]
  node_bigcode_workflow["BigCode flow<br/>workflow<br/>[workflow.py]"]
  node_bigcode_eval["BigCode eval<br/>task eval<br/>[eval.py]"]
end

subgraph group_metrics["Metrics"]
  node_metrics_f1(("F1<br/>metric<br/>[f1.py]"))
  node_metrics_mrr(("MRR<br/>metric<br/>[mrr.py]"))
  node_metrics_pass(("Pass rate<br/>metric<br/>[pass_rate.py]"))
end

subgraph group_assets["Assets"]
  node_configs["Configs<br/>experiment config"]
  node_pubmed_data_dir[("PubMed files<br/>local data")]
  node_bigcode_assets["BigCode assets<br/>task assets"]
  node_vendor_textgrad["TextGrad<br/>vendored runtime"]
  node_external_llm(("LLM gateway<br/>external service"))
  node_external_datasets(("Dataset hubs<br/>external service"))
  node_external_stark(("stark_qa<br/>external package"))
end

node_cli_train -->|"starts"| node_runtime_core
node_cli_evaluate -->|"scores"| node_runtime_core
node_cli_results -->|"orchestrates"| node_runtime_core
node_cli_manifest -->|"builds"| node_split_utils
node_runtime_core -->|"updates"| node_prompt_state
node_runtime_core -->|"resolves"| node_registry
node_runtime_core -->|"calls"| node_external_llm
node_runtime_core -->|"uses"| node_vendor_textgrad
node_registry -->|"loads"| node_hotpot_workflow
node_registry -->|"loads"| node_pubmed_workflow
node_registry -->|"loads"| node_stark_workflow
node_registry -->|"loads"| node_bigcode_workflow
node_hotpot_workflow -->|"reads"| node_hotpot_dataset
node_hotpot_dataset -->|"sources"| node_external_datasets
node_pubmed_workflow -->|"reads"| node_pubmed_dataset
node_pubmed_dataset -->|"loads"| node_pubmed_data_dir
node_pubmed_workflow -->|"delegates"| node_pubmed_eval
node_pubmed_eval -->|"scores"| node_metrics_f1
node_stark_workflow -->|"reads"| node_stark_dataset
node_stark_dataset -->|"depends on"| node_external_stark
node_stark_workflow -->|"scores"| node_metrics_mrr
node_bigcode_workflow -->|"reads"| node_bigcode_dataset
node_bigcode_dataset -->|"sources"| node_external_datasets
node_bigcode_workflow -->|"delegates"| node_bigcode_eval
node_bigcode_eval -->|"scores"| node_metrics_pass
node_hotpot_workflow -->|"scores"| node_metrics_f1
node_configs -->|"configures"| node_cli_train
node_configs -->|"configures"| node_cli_evaluate
node_configs -->|"configures"| node_cli_results
node_configs -->|"configures"| node_cli_manifest
node_bigcode_assets -->|"initializes"| node_bigcode_dataset
node_bigcode_assets -->|"feeds"| node_split_utils

click node_cli_train "https://github.com/minghuichen43/tep/blob/main/src/tep/cli/train.py"
click node_cli_evaluate "https://github.com/minghuichen43/tep/blob/main/src/tep/cli/evaluate.py"
click node_cli_results "https://github.com/minghuichen43/tep/blob/main/src/tep/cli/run_main_results.py"
click node_cli_manifest "https://github.com/minghuichen43/tep/blob/main/src/tep/cli/build_split_manifest.py"
click node_runtime_core "https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/core.py"
click node_prompt_state "https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/prompt_state.py"
click node_registry "https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/registry.py"
click node_split_utils "https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/split_utils.py"
click node_hotpot_dataset "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/hotpotqa/dataset.py"
click node_hotpot_workflow "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/hotpotqa/workflow.py"
click node_pubmed_dataset "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/dataset.py"
click node_pubmed_workflow "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/workflow.py"
click node_pubmed_eval "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/eval.py"
click node_stark_dataset "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/stark/dataset.py"
click node_stark_workflow "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/stark/workflow.py"
click node_bigcode_dataset "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/dataset.py"
click node_bigcode_workflow "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/workflow.py"
click node_bigcode_eval "https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/eval.py"
click node_metrics_f1 "https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/f1.py"
click node_metrics_mrr "https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/mrr.py"
click node_metrics_pass "https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/pass_rate.py"
click node_configs "https://github.com/minghuichen43/tep/tree/main/configs"
click node_pubmed_data_dir "https://github.com/minghuichen43/tep/tree/main/data/pubmed"
click node_bigcode_assets "https://github.com/minghuichen43/tep/tree/main/src/tep/tasks/bigcodebench/assets"
click node_vendor_textgrad "https://github.com/minghuichen43/tep/tree/main/src/tep/vendor"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_train,node_cli_evaluate,node_cli_results,node_cli_manifest toneBlue
class node_runtime_core,node_prompt_state,node_registry,node_split_utils toneAmber
class node_hotpot_dataset,node_hotpot_workflow,node_pubmed_dataset,node_pubmed_workflow,node_pubmed_eval,node_stark_dataset,node_stark_workflow,node_bigcode_dataset,node_bigcode_workflow,node_bigcode_eval toneMint
class node_metrics_f1,node_metrics_mrr,node_metrics_pass toneRose
class node_configs,node_pubmed_data_dir,node_bigcode_assets,node_vendor_textgrad,node_external_llm,node_external_datasets,node_external_stark toneIndigo
```

## Components

### CLI

User-facing experiment commands

- [Train](https://github.com/minghuichen43/tep/blob/main/src/tep/cli/train.py): cli · Runs TEP optimization · `src/tep/cli/train.py`
- [Evaluate](https://github.com/minghuichen43/tep/blob/main/src/tep/cli/evaluate.py): cli · Scores a configured workflow · `src/tep/cli/evaluate.py`
- [Main results](https://github.com/minghuichen43/tep/blob/main/src/tep/cli/run_main_results.py): cli · Runs multi-benchmark experiments · `src/tep/cli/run_main_results.py`
- [Split manifest](https://github.com/minghuichen43/tep/blob/main/src/tep/cli/build_split_manifest.py): cli · Builds dataset split manifests · `src/tep/cli/build_split_manifest.py`

### Runtime

Shared optimization and orchestration

- [Core loop](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/core.py): runtime · Free-phase and nudged-phase execution loop · `src/tep/runtime/core.py`
- [Prompt state](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/prompt_state.py): runtime state · Mutable prompt/workflow state across phases · `src/tep/runtime/prompt_state.py`
- [Registry](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/registry.py): runtime registry · Lookup and registration boundary · `src/tep/runtime/registry.py`
- [Split utils](https://github.com/minghuichen43/tep/blob/main/src/tep/runtime/split_utils.py): runtime splits · Dataset split and manifest handling · `src/tep/runtime/split_utils.py`

### Tasks

Benchmark-specific datasets and workflows

- [Hotpot data](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/hotpotqa/dataset.py): dataset · HotpotQA data loading · `src/tep/tasks/hotpotqa/dataset.py`
- [Hotpot flow](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/hotpotqa/workflow.py): workflow · Multi-hop QA workflow composition · `src/tep/tasks/hotpotqa/workflow.py`
- [PubMed data](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/dataset.py): dataset · Local PubMedQA JSONL loading · `src/tep/tasks/pubmed/dataset.py`
- [PubMed flow](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/workflow.py): workflow · Biomedical QA workflow composition · `src/tep/tasks/pubmed/workflow.py`
- [PubMed eval](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/pubmed/eval.py): task eval · Task-specific PubMed evaluation helpers · `src/tep/tasks/pubmed/eval.py`
- [STARK data](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/stark/dataset.py): dataset · STARK-PRIME retrieval data loading · `src/tep/tasks/stark/dataset.py`
- [STARK flow](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/stark/workflow.py): workflow · Retrieval-style QA workflow composition · `src/tep/tasks/stark/workflow.py`
- [BigCode data](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/dataset.py): dataset · BigCodeBench benchmark loading · `src/tep/tasks/bigcodebench/dataset.py`
- [BigCode flow](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/workflow.py): workflow · Code generation and repair workflow · `src/tep/tasks/bigcodebench/workflow.py`
- [BigCode eval](https://github.com/minghuichen43/tep/blob/main/src/tep/tasks/bigcodebench/eval.py): task eval · Task-specific code evaluation helpers · `src/tep/tasks/bigcodebench/eval.py`

### Metrics

Reusable scoring functions

- [F1](https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/f1.py): metric · Overlap-style QA scoring · `src/tep/metrics/f1.py`
- [MRR](https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/mrr.py): metric · Ranking quality scoring · `src/tep/metrics/mrr.py`
- [Pass rate](https://github.com/minghuichen43/tep/blob/main/src/tep/metrics/pass_rate.py): metric · Code-task pass/fail scoring · `src/tep/metrics/pass_rate.py`

### Assets

Configs, local data, and vendored runtime

- [Configs](https://github.com/minghuichen43/tep/tree/main/configs): experiment config · Task-specific experiment YAMLs · `configs`
- [PubMed files](https://github.com/minghuichen43/tep/tree/main/data/pubmed): local data · PubMedQA train/test JSONL files · `data/pubmed`
- [BigCode assets](https://github.com/minghuichen43/tep/tree/main/src/tep/tasks/bigcodebench/assets): task assets · Prompt init and fixed split manifest · `src/tep/tasks/bigcodebench/assets`
- [TextGrad](https://github.com/minghuichen43/tep/tree/main/src/tep/vendor): vendored runtime · Internal prompt-optimization primitives · `src/tep/vendor`
- **LLM gateway**: external service · Remote model access via environment configuration
- **Dataset hubs**: external service · HuggingFace and other external data sources
- **stark_qa**: external package · STARK benchmark dependency

## Connections

- Train → Core loop: starts · Launches the optimization loop
- Evaluate → Core loop: scores · Runs configured evaluation through the shared runtime
- Main results → Core loop: orchestrates · Drives repeated benchmark executions
- Split manifest → Split utils: builds · Produces or manages split manifests
- Core loop → Prompt state: updates · Advances prompt state across free and nudged phases
- Core loop → Registry: resolves · Finds workflows and task components
- Core loop → LLM gateway: calls · Uses remote model inference
- Core loop → TextGrad: uses · Relies on vendored prompt optimization primitives
- Registry → Hotpot flow: loads · Registers benchmark workflow
- Registry → PubMed flow: loads · Registers benchmark workflow
- Registry → STARK flow: loads · Registers benchmark workflow
- Registry → BigCode flow: loads · Registers benchmark workflow
- Hotpot flow → Hotpot data: reads · Consumes HotpotQA examples
- Hotpot data → Dataset hubs: sources · Loads benchmark data from external datasets
- PubMed flow → PubMed data: reads · Consumes local PubMedQA data
- PubMed data → PubMed files: loads · Reads JSONL files from the repository
- PubMed flow → PubMed eval: delegates · Uses task-specific PubMed evaluation helpers
- PubMed eval → F1: scores · Evaluates QA overlap
- STARK flow → STARK data: reads · Consumes STARK-PRIME data
- STARK data → stark_qa: depends on · Uses the external STARK package
- STARK flow → MRR: scores · Uses ranking metric for retrieval-style evaluation
- BigCode flow → BigCode data: reads · Consumes BigCodeBench data
- BigCode data → Dataset hubs: sources · Loads benchmark data from external datasets
- BigCode flow → BigCode eval: delegates · Uses task-specific code evaluation helpers
- BigCode eval → Pass rate: scores · Evaluates pass/fail outcomes
- Hotpot flow → F1: scores · Uses QA overlap scoring
- Configs → Train: configures · Task experiment settings
- Configs → Evaluate: configures · Task experiment settings
- Configs → Main results: configures · Benchmark run settings
- Configs → Split manifest: configures · Split-manifest settings
- BigCode assets → BigCode data: initializes · Provides prompt init and split manifest
- BigCode assets → Split utils: feeds · Supplies reproducible split metadata
