How vita-group/o1-planning works
Project Type and Purpose
• This is an open-source research evaluation pipeline that measures the planning abilities of OpenAI’s GPT-4 and o1 models on PDDL tasks.
• It ingests PDDL domains and problems, generates natural-language prompts, calls OpenAI’s API, stores the raw outputs, and produces visual analyses.
High-Level Components
a. Data Sources
• PDDL domain/problem definitions (hard-coded in prompts under results/<domain>/p*.pddl.prompt)
• Historical runs (results/<domain>/.history)
b. Prompt-Generation & Automation
• Python scripts in scripts/ (e.g., create_empty_files.py, derandom.py, random.py)
• They assemble prompt files and invoke the LLM clients
c. LLM API Clients
• Integrations to OpenAI’s GPT-4 and o1 models (o1-mini, o1-preview)
• Handles authentication, rate limiting, JSON I/O
d. Results Storage
• File-based repository under results/
• Organized by domain (e.g., barman, blocksworld, tyreworld) and by model output type (.gpt4, .o1-mini, .o1-preview)
e. Evaluation & Visualization
• Google Sheets for aggregated metrics (linked in README)
• Static graphics in visual/ (radar_chart.pdf, .png)
• Future graphing scripts can be added here
Relationships & Data Flows
1. Script picks up domain/problem → generates .prompt file
2. Script calls LLM API → receives .gpt4 / .o1-* responses
3. Responses saved in results/<domain>/ alongside original prompt
4. (Optionally) historical snapshots archived under results/<domain>/.history
5. Aggregation script (external or manual) reads all results/ files → writes metrics to Google Sheets
6. Visualization tools pull from Sheets or local metrics → produce charts in visual/
Architectural Patterns & Principles
• Pipeline/ETL pattern: input data → transform (prompt) → load (model) → extract (store) → analyze
• Modular scripts: each script focuses on one domain or one function (randomization, derandomization, file skeleton)
• File-based, domain-centric organization: simple, reproducible experiment runs
• Separation of concerns: prompt generation vs API invocation vs visualization
Essential Details to Capture
• Historical capture: include the .history folder as a versioning/archive sub-component of Results Store
• Scripts specialization: call out random.py/derandom.py for randomized tests
• Visual outputs: point to visual/ and show that charts derive from aggregated results