How mikewangwzhl/solo-performance-prompting works
Project type and purpose
• This repo is an open-source experimental framework (not a user-facing full-stack app). It implements the Solo Performance Prompting (SPP) methodology to compare different prompting strategies (standard, chain-of-thought, SPP, self-refine) on three NLP tasks (Codenames, Logic Grid Puzzle, Trivia Creative Writing).
• It orchestrates data loading, prompt construction, LLM API calls (OpenAI/Azure/LLama), response parsing, metric computation, and logging.
File structure
• Top-level orchestration:
• run.py – main entry point
• scripts/*.sh – bash wrappers invoking run.py with task-specific args
• Configuration:
• config_template.sh / configs.py – environment variables, API keys, hyperparameters
• Core modules:
• models.py – wrapper around LLM engines (OpenAI/Azure/LLama)
• prompts/ – prompt templates per task
• tasks/ – Task classes inheriting from tasks/base.py; each loads its dataset, applies prompts, calls model, parses & evaluates
• Data & outputs:
• data/ – JSONL or TXT files with task datasets
• logs/ – hierarchical folders by task/engine/method, containing raw responses, parsing flags, metrics, token usage
• Utilities:
• scripts/ – shell scripts for quick runs
• requirements.txt – Python dependencies
README for additional insights
• Describes the overall pipeline: datasets → prompts → API calls → logs (metrics, raw responses)
• Mentions usage of Azure OpenAI or other LLM endpoints
• Explains the log file format (fields like prompt, raw_responses, parsing_flag, unwrapped_output, usage, etc.)
Main components & interactions
a. Components
• CLI Orchestrator (run.py + scripts/*.sh)
• Configuration Module (config_template.sh + configs.py)
• Data Storage (data/ folder)
• Prompt Builder (prompts/ folder)
• Task Executor (tasks/base.py + tasks/*.py)
• Model Service Wrapper (models.py → external LLM APIs)
• Response Parser & Evaluator (inside Task Executor)
• Log Storage (logs/ folder)
b. Relationships & data flow
1. User runs a shell script (e.g., scripts/codenames_collaborative.sh), which:
– Sources config_template.sh → exports env vars
– Calls run.py with task name, engine, method, hyperparams
2. run.py reads configs.py + env vars → initializes TaskExecutor for the selected task
3. TaskExecutor loads dataset from data/ → for each instance:
– Passes instance data to PromptBuilder to generate full prompt(s)
– Invokes ModelServiceWrapper to send prompt → external LLM API → receives raw response
– ResponseParser extracts structured output (parsing_flag, unwrapped_output)
4. TaskExecutor computes metrics (accuracy, success flags)
5. All intermediate (prompt, raw_responses, parsing_flag, usage) & final metrics are written into logs/ under appropriate subfolder
6. End of run: summary metrics printed to console or saved alongside logs
c. Architectural patterns & design principles
• Modular design with clear separation of concerns:
• “prompts” purely holds text templates
• “tasks” implements task logic, dataset handling, parsing, evaluation
• “models.py” isolates API details from business logic
• Extensible: new tasks can be added by subclassing TaskBase and placing prompts in prompts/
• Scriptable CLI: allows batch runs, hyperparameter sweeps via shell scripts
d. Key technologies & frameworks
– Python 3
– OpenAI/Azure LLM API, HuggingFace “meta-llama” client
– JSONL for data & logs
– Bash scripts for orchestration
– Standard libraries (argparse, logging), plus any dependencies in requirements.txt