How foundationagents/spo works
Project type and purpose
• This is a Python-based prompt engineering tool (“SPO”) for self-supervised prompt optimization on large language models (LLMs).
• It can be driven via a CLI (optimize.py), a Python API (components/optimizer.py), or a Streamlit web UI (app.py).
• It reads iteration templates (settings/*.yaml), optional data (data/*/*.jsonl), and LLM configuration (config/*.yaml), then repeatedly calls LLMs for optimization, evaluation, and execution, producing optimized prompts and answers in a workspace directory.
File structure
• Top-level entry points:
• optimize.py (CLI runner)
• app.py (Streamlit UI)
• Core logic:
• components/optimizer.py – orchestrates optimization loops
• components/evaluator.py – judges LLM outputs
• LLM interface:
• llm/async_llm.py – async wrappers around LLM API calls
• utils/llm_client.py – high-level client managing optimize/evaluate/execute calls
• Configuration & templates:
• config/ – sample LLM API settings
• settings/ – iteration templates defining prompts, requirements, QA pairs
• Prompts:
• prompts/evaluate_prompt.py, prompts/optimize_prompt.py – prompt engineering helpers
• Data utilities:
• utils/data_utils.py, utils/prompt_utils.py, utils/evaluation_utils.py, utils/load.py
• Datasets:
• data/* – training/validation/test JSONL files for various benchmarks (MATH, BBH, GPQA, etc.)
• Requirements & assets:
• requirements.txt, static images under assert/, and LICENSE
README for additional insights
• Confirms three operation modes: Python API, CLI, Streamlit UI.
• Workflow:
• Initialize LLM settings → create PromptOptimizer → call optimize() → generate workspace with per‐round prompt/answers and results.json.
• Highlights external LLM services (Claude, GPT-4) as “judge” and “executor.”