# malusamayo/underspec-analysis architecture

> Architecture diagram of the GitHub repository malusamayo/underspec-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/malusamayo/underspec-analysis
- Repository: https://github.com/malusamayo/underspec-analysis

## Overview

This is a research reproduction toolkit for studying how prompt underspecification affects LLM outputs and evaluation. The main workflow loads task data, requirements, and prompts; runs configured model/prompt experiments; evaluates outputs against requirements; and saves evaluation results for analysis. Separate research branches generate requirements and evaluators, generate prompts, and optimize prompts. The graph preserves these documented capabilities; some wiring is supported only by the README or source index, so unsampled implementation details are not inferred.

Main entry points: [`analysis/optimize.py`](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimize.py) (Prompt optimization), [`analysis/load_data.py`](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/load_data.py) (Task data loader), [`run.py`](https://github.com/malusamayo/underspec-analysis/blob/main/run.py) (Run experiments), [`analysis/judge.py`](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/judge.py) (Output judge)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experiment["Experiment execution"]
  node_run["Run experiments<br/>[run.py]"]
  node_taskdata["Task data loader<br/>[load_data.py]"]
  node_models["LLM providers"]
  node_resultfiles["Evaluation results"]
end

subgraph group_evaluation["Evaluation and analysis"]
  node_judge["Output judge<br/>[judge.py]"]
  node_agent_eval["Agent evaluator<br/>[agent_eval.py]"]
  node_analysisutils["Model utilities<br/>[utils.py]"]
end

subgraph group_prompting["Prompt research"]
  node_requirements["Requirement elicitation<br/>[elicitation.py]"]
  node_evalgen["Evaluator generation<br/>[judge.py]"]
  node_promptgen["Prompt generation<br/>[prompt_gen.py]"]
  node_optimize["Prompt optimization<br/>[optimize.py]"]
  node_copro["COPRO optimizer<br/>[copro.py]"]
  node_textgrad["TextGrad optimizer<br/>[textgrad.py]"]
  node_reqaware["Requirement optimizer<br/>[reqaware.py]"]
  node_openaiopt["Prompt optimizer<br/>[openai.py]"]
end

subgraph group_resources["Experiment resources"]
  node_configs["Experiment configs"]
  node_prompts["Prompt sets"]
  node_reqdata["Curated requirements"]
  node_datasets["Task datasets"]
end

node_researcher(("Researcher"))

node_researcher -->|"configures"| node_run
node_run -.->|"reads config"| node_configs
node_run -->|"loads data"| node_taskdata
node_run -->|"evaluates outputs"| node_judge
node_taskdata -->|"requests completions"| node_models
node_run -->|"writes results"| node_resultfiles
node_taskdata -.->|"loads task data"| node_datasets
node_taskdata -.->|"loads prompts"| node_prompts
node_taskdata -.->|"loads requirements"| node_reqdata
node_judge -->|"uses inference"| node_analysisutils
node_judge -.->|"scores outputs"| node_models
node_agent_eval -.->|"evaluates responses"| node_models
node_researcher -->|"elicits requirements"| node_requirements
node_requirements -->|"uses judge"| node_judge
node_requirements -->|"uses helpers"| node_analysisutils
node_researcher -->|"generates evaluators"| node_evalgen
node_researcher -->|"generates prompts"| node_promptgen
node_researcher -->|"optimizes prompts"| node_optimize
node_optimize -->|"prepares data"| node_taskdata
node_optimize -->|"evaluates candidates"| node_judge
node_optimize -->|"selects optimizer"| node_copro
node_optimize -->|"selects optimizer"| node_textgrad
node_optimize -->|"selects optimizer"| node_reqaware
node_optimize -->|"selects optimizer"| node_openaiopt
node_optimize -.->|"uses prompt sets"| node_prompts
node_optimize -.->|"uses requirements"| node_reqdata

click node_run "https://github.com/malusamayo/underspec-analysis/blob/main/run.py"
click node_taskdata "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/load_data.py"
click node_judge "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/judge.py"
click node_agent_eval "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/agent_eval.py"
click node_requirements "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/elicitation.py"
click node_evalgen "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/judge.py"
click node_promptgen "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/prompt_gen.py"
click node_optimize "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimize.py"
click node_copro "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/copro.py"
click node_textgrad "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/textgrad.py"
click node_reqaware "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/reqaware.py"
click node_openaiopt "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/openai.py"
click node_configs "https://github.com/malusamayo/underspec-analysis/tree/main/data/configs"
click node_prompts "https://github.com/malusamayo/underspec-analysis/tree/main/data/prompts"
click node_reqdata "https://github.com/malusamayo/underspec-analysis/tree/main/data/requirements"
click node_datasets "https://github.com/malusamayo/underspec-analysis/tree/main/data"
click node_analysisutils "https://github.com/malusamayo/underspec-analysis/blob/main/analysis/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_run,node_taskdata,node_models,node_resultfiles toneBlue
class node_judge,node_agent_eval,node_analysisutils toneAmber
class node_requirements,node_evalgen,node_promptgen,node_optimize,node_copro,node_textgrad,node_reqaware,node_openaiopt toneMint
class node_configs,node_prompts,node_reqdata,node_datasets toneRose
class node_researcher toneIndigo
```

## Components

### Experiment execution

- [Run experiments](https://github.com/malusamayo/underspec-analysis/blob/main/run.py): `run.py`
- [Task data loader](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/load_data.py): `analysis/load_data.py`
- **LLM providers**
- **Evaluation results**

### Evaluation and analysis

- [Output judge](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/judge.py): `analysis/judge.py`
- [Agent evaluator](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/agent_eval.py): `analysis/agent_eval.py`
- [Model utilities](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/utils.py): `analysis/utils.py`

### Prompt research

- [Requirement elicitation](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/elicitation.py): `analysis/elicitation.py`
- [Evaluator generation](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/judge.py): `analysis/judge.py`
- [Prompt generation](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/prompt_gen.py): `analysis/prompt_gen.py`
- [Prompt optimization](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimize.py): `analysis/optimize.py`
- [COPRO optimizer](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/copro.py): `analysis/optimizers/copro.py`
- [TextGrad optimizer](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/textgrad.py): `analysis/optimizers/textgrad.py`
- [Requirement optimizer](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/reqaware.py): `analysis/optimizers/reqaware.py`
- [Prompt optimizer](https://github.com/malusamayo/underspec-analysis/blob/main/analysis/optimizers/openai.py): `analysis/optimizers/openai.py`

### Experiment resources

- [Experiment configs](https://github.com/malusamayo/underspec-analysis/tree/main/data/configs): `data/configs`
- [Prompt sets](https://github.com/malusamayo/underspec-analysis/tree/main/data/prompts): `data/prompts`
- [Curated requirements](https://github.com/malusamayo/underspec-analysis/tree/main/data/requirements): `data/requirements`
- [Task datasets](https://github.com/malusamayo/underspec-analysis/tree/main/data): `data`

### Other components

- **Researcher**

## Connections

- Researcher → Run experiments: configures (evidence: `README.md`)
- Run experiments → Experiment configs: reads config
- Run experiments → Task data loader: loads data (evidence: `run.py`)
- Run experiments → Output judge: evaluates outputs (evidence: `run.py`)
- Task data loader → LLM providers: requests completions (evidence: `analysis/load_data.py`)
- Run experiments → Evaluation results: writes results (evidence: `run.py`)
- Task data loader → Task datasets: loads task data
- Task data loader → Prompt sets: loads prompts
- Task data loader → Curated requirements: loads requirements
- Output judge → Model utilities: uses inference (evidence: `analysis/judge.py`)
- Output judge → LLM providers: scores outputs
- Agent evaluator → LLM providers: evaluates responses
- Researcher → Requirement elicitation: elicits requirements (evidence: `README.md`)
- Requirement elicitation → Output judge: uses judge (evidence: `analysis/elicitation.py`)
- Requirement elicitation → Model utilities: uses helpers (evidence: `analysis/elicitation.py`)
- Researcher → Evaluator generation: generates evaluators (evidence: `README.md`)
- Researcher → Prompt generation: generates prompts (evidence: `README.md`)
- Researcher → Prompt optimization: optimizes prompts (evidence: `README.md`)
- Prompt optimization → Task data loader: prepares data (evidence: `analysis/optimize.py`)
- Prompt optimization → Output judge: evaluates candidates (evidence: `analysis/optimize.py`)
- Prompt optimization → COPRO optimizer: selects optimizer (evidence: `analysis/optimize.py`)
- Prompt optimization → TextGrad optimizer: selects optimizer (evidence: `analysis/optimize.py`)
- Prompt optimization → Requirement optimizer: selects optimizer (evidence: `analysis/optimize.py`)
- Prompt optimization → Prompt optimizer: selects optimizer (evidence: `analysis/optimize.py`)
- Prompt optimization → Prompt sets: uses prompt sets
- Prompt optimization → Curated requirements: uses requirements
