# mikewangwzhl/solo-performance-prompting architecture

> Architecture diagram of the GitHub repository mikewangwzhl/solo-performance-prompting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mikewangwzhl/solo-performance-prompting
- Repository: https://github.com/mikewangwzhl/solo-performance-prompting

## Overview

##### Project type and purpose
- This repo is an open-source experimental framework (not a user-facing full-stack app). It implements the Solo Performance Prompting (SPP) methodology to compare different prompting strategies (standard, chain-of-thought, SPP, self-refine) on three NLP tasks (Codenames, Logic Grid Puzzle, Trivia Creative Writing).
- It orchestrates data loading, prompt construction, LLM API calls (OpenAI/Azure/LLama), response parsing, metric computation, and logging.

##### File structure
- Top-level orchestration:
- run.py – main entry point
- scripts/*.sh – bash wrappers invoking run.py with task-specific args
- Configuration:
- config_template.sh / configs.py – environment variables, API keys, hyperparameters
- Core modules:
- models.py – wrapper around LLM engines (OpenAI/Azure/LLama)
- prompts/ – prompt templates per task
- tasks/ – Task classes inheriting from tasks/base.py; each loads its dataset, applies prompts, calls model, parses & evaluates
- Data & outputs:
- data/ – JSONL or TXT files with task datasets
- logs/ – hierarchical folders by task/engine/method, containing raw responses, parsing flags, metrics, token usage
- Utilities:
- scripts/ – shell scripts for quick runs
- requirements.txt – Python dependencies

##### README for additional insights
- Describes the overall pipeline: datasets → prompts → API calls → logs (metrics, raw responses)
- Mentions usage of Azure OpenAI or other LLM endpoints
- Explains the log file format (fields like prompt, raw_responses, parsing_flag, unwrapped_output, usage, etc.)

##### Main components & interactions
a. Components
- CLI Orchestrator (run.py + scripts/*.sh)
- Configuration Module (config_template.sh + configs.py)
- Data Storage (data/ folder)
- Prompt Builder (prompts/ folder)
- Task Executor (tasks/base.py + tasks/*.py)
- Model Service Wrapper (models.py → external LLM APIs)
- Response Parser & Evaluator (inside Task Executor)
- Log Storage (logs/ folder)

b. Relationships & data flow
1. User runs a shell script (e.g., scripts/codenames_collaborative.sh), which:
– Sources config_template.sh → exports env vars
– Calls run.py with task name, engine, method, hyperparams
2. run.py reads configs.py + env vars → initializes TaskExecutor for the selected task
3. TaskExecutor loads dataset from data/ → for each instance:
– Passes instance data to PromptBuilder to generate full prompt(s)
– Invokes ModelServiceWrapper to send prompt → external LLM API → receives raw response
– ResponseParser extracts structured output (parsing_flag, unwrapped_output)
4. TaskExecutor computes metrics (accuracy, success flags)
5. All intermediate (prompt, raw_responses, parsing_flag, usage) & final metrics are written into logs/ under appropriate subfolder
6. End of run: summary metrics printed to console or saved alongside logs

c. Architectural patterns & design principles
- Modular design with clear separation of concerns:
- “prompts” purely holds text templates
- “tasks” implements task logic, dataset handling, parsing, evaluation
- “models.py” isolates API details from business logic
- Extensible: new tasks can be added by subclassing TaskBase and placing prompts in prompts/
- Scriptable CLI: allows batch runs, hyperparameter sweeps via shell scripts

d. Key technologies & frameworks
– Python 3
– OpenAI/Azure LLM API, HuggingFace “meta-llama” client
– JSONL for data & logs
– Bash scripts for orchestration
– Standard libraries (argparse, logging), plus any dependencies in requirements.txt

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "CLI & Scripts"
        direction TB
        A["run.py"]:::python
        B1["scripts/codenames_collaborative.sh"]:::script
        B2["scripts/logic_grid_puzzle.sh"]:::script
        B3["scripts/trivia_creative_writing.sh"]:::script
    end

    subgraph "Configuration Module"
        direction TB
        C["config_template.sh"]:::script
        D["configs.py"]:::python
    end

    subgraph "Data Storage"
        direction TB
        E["data/"]:::data
    end

    subgraph "Backend"
        direction TB
        subgraph "Prompt Builder"
            direction TB
            F["prompts/codenames_collaborative.py"]:::python
            G["prompts/logic_grid_puzzle.py"]:::python
            H["prompts/trivia_creative_writing.py"]:::python
        end
        subgraph "Task Executor"
            direction TB
            I["tasks/base.py"]:::python
            J["tasks/codenames_collaborative.py"]:::python
            K["tasks/logic_grid_puzzle.py"]:::python
            L["tasks/trivia_creative_writing.py"]:::python
        end
        M["models.py"]:::python
    end

    N["External LLM API (OpenAI/Azure/LLama)"]:::external
    O["Response Parser & Evaluator"]:::python

    subgraph "Logs Storage"
        direction TB
        P["logs/"]:::data
    end

    %% Connections
    B1 -->|"calls"| A
    B2 -->|"calls"| A
    B3 -->|"calls"| A
    A -->|"read env & config"| D
    A -->|"initialize TaskExecutor"| I
    I -->|"load dataset"| E
    E -->|"JSONL instance data"| F
    F -->|"prompt text"| M
    M -->|"HTTP request"| N
    N -->|"raw JSON response"| M
    M -->|"raw response"| O
    O -->|"parsed_output"| I
    I -->|"metrics"| P
    O -->|"write logs"| P

    %% Click Events
    click A "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/run.py"
    click B1 "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/scripts/codenames_collaborative.sh"
    click B2 "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/scripts/logic_grid_puzzle.sh"
    click B3 "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/scripts/trivia_creative_writing.sh"
    click C "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/config_template.sh"
    click D "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/configs.py"
    click E "https://github.com/mikewangwzhl/solo-performance-prompting/tree/main/data/"
    click F "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/prompts/codenames_collaborative.py"
    click G "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/prompts/logic_grid_puzzle.py"
    click H "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/prompts/trivia_creative_writing.py"
    click I "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/tasks/base.py"
    click J "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/tasks/codenames_collaborative.py"
    click K "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/tasks/logic_grid_puzzle.py"
    click L "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/tasks/trivia_creative_writing.py"
    click M "https://github.com/mikewangwzhl/solo-performance-prompting/blob/main/models.py"
    click P "https://github.com/mikewangwzhl/solo-performance-prompting/tree/main/logs/"

    %% Styles
    classDef python fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef script fill:#FFFF66,stroke:#000,stroke-width:1px;
    classDef data fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px;
```
