How malusamayo/underspec-analysis works
This is a research reproduction toolkit for studying how prompt underspecification affects LLM outputs and evaluation. The main workflow loads task data, requirements, and prompts; runs configured model/prompt experiments; evaluates outputs against requirements; and saves evaluation results for analysis. Separate research branches generate requirements and evaluators, generate prompts, and optimize prompts. The graph preserves these documented capabilities; some wiring is supported only by the README or source index, so unsampled implementation details are not inferred.
Main entry points: optimize.py Prompt optimization · load_data.py Task data loader · run.py Run experiments · judge.py Output judge
Components 20
Experiment execution
- Run experiments
run.py - Task data loader
analysis/load_data.py - LLM providers
- Evaluation results
Evaluation and analysis
- Output judge
analysis/judge.py - Agent evaluator
analysis/agent_eval.py - Model utilities
analysis/utils.py
Prompt research
- Requirement elicitation
analysis/elicitation.py - Evaluator generation
analysis/judge.py - Prompt generation
analysis/prompt_gen.py - Prompt optimization
analysis/optimize.py - COPRO optimizer
analysis/optimizers/copro.py - TextGrad optimizer
analysis/optimizers/textgrad.py - Requirement optimizer
analysis/optimizers/reqaware.py - Prompt optimizer
analysis/optimizers/openai.py
Experiment resources
- Experiment configs
data/configs - Prompt sets
data/prompts - Curated requirements
data/requirements - Task datasets
data
Other components
- Researcher
Connections 26
- Researcher → Run experiments: configures
README.md - Run experiments → Experiment configs: reads config
- Run experiments → Task data loader: loads data
run.py - Run experiments → Output judge: evaluates outputs
run.py - Task data loader → LLM providers: requests completions
analysis/load_data.py - Run experiments → Evaluation results: writes results
run.py - Task data loader → Task datasets: loads task data
- Task data loader → Prompt sets: loads prompts
- Task data loader → Curated requirements: loads requirements
- Output judge → Model utilities: uses inference
analysis/judge.py - Output judge → LLM providers: scores outputs
- Agent evaluator → LLM providers: evaluates responses
- Researcher → Requirement elicitation: elicits requirements
README.md - Requirement elicitation → Output judge: uses judge
analysis/elicitation.py - Requirement elicitation → Model utilities: uses helpers
analysis/elicitation.py - Researcher → Evaluator generation: generates evaluators
README.md - Researcher → Prompt generation: generates prompts
README.md - Researcher → Prompt optimization: optimizes prompts
README.md - Prompt optimization → Task data loader: prepares data
analysis/optimize.py - Prompt optimization → Output judge: evaluates candidates
analysis/optimize.py - Prompt optimization → COPRO optimizer: selects optimizer
analysis/optimize.py - Prompt optimization → TextGrad optimizer: selects optimizer
analysis/optimize.py - Prompt optimization → Requirement optimizer: selects optimizer
analysis/optimize.py - Prompt optimization → Prompt optimizer: selects optimizer
analysis/optimize.py - Prompt optimization → Prompt sets: uses prompt sets
- Prompt optimization → Curated requirements: uses requirements