# vita-group/o1-planning architecture

> Architecture diagram of the GitHub repository vita-group/o1-planning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vita-group/o1-planning
- Repository: https://github.com/vita-group/o1-planning

## Overview

##### Project Type and Purpose
- This is an open-source research evaluation pipeline that measures the planning abilities of OpenAI’s GPT-4 and o1 models on PDDL tasks.
- It ingests PDDL domains and problems, generates natural-language prompts, calls OpenAI’s API, stores the raw outputs, and produces visual analyses.

##### High-Level Components
a. Data Sources
- PDDL domain/problem definitions (hard-coded in prompts under `results/<domain>/p*.pddl.prompt`)
- Historical runs (`results/<domain>/.history`)
b. Prompt-Generation & Automation
- Python scripts in `scripts/` (e.g., `create_empty_files.py`, `derandom.py`, `random.py`)
- They assemble prompt files and invoke the LLM clients
c. LLM API Clients
- Integrations to OpenAI’s GPT-4 and o1 models (o1-mini, o1-preview)
- Handles authentication, rate limiting, JSON I/O
d. Results Storage
- File-based repository under `results/`
- Organized by domain (e.g., barman, blocksworld, tyreworld) and by model output type (`.gpt4`, `.o1-mini`, `.o1-preview`)
e. Evaluation & Visualization
- Google Sheets for aggregated metrics (linked in README)
- Static graphics in `visual/` (`radar_chart.pdf`, `.png`)
- Future graphing scripts can be added here

##### Relationships & Data Flows
1. Script picks up domain/problem → generates `.prompt` file
2. Script calls LLM API → receives `.gpt4` / `.o1-*` responses
3. Responses saved in `results/<domain>/` alongside original prompt
4. (Optionally) historical snapshots archived under `results/<domain>/.history`
5. Aggregation script (external or manual) reads all `results/` files → writes metrics to Google Sheets
6. Visualization tools pull from Sheets or local metrics → produce charts in `visual/`

##### Architectural Patterns & Principles
- Pipeline/ETL pattern: input data → transform (prompt) → load (model) → extract (store) → analyze
- Modular scripts: each script focuses on one domain or one function (randomization, derandomization, file skeleton)
- File-based, domain-centric organization: simple, reproducible experiment runs
- Separation of concerns: prompt generation vs API invocation vs visualization

##### Essential Details to Capture
- Historical capture: include the `.history` folder as a versioning/archive sub-component of Results Store
- Scripts specialization: call out `random.py`/`derandom.py` for randomized tests
- Visual outputs: point to `visual/` and show that charts derive from aggregated results

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        PDDL["PDDL Domains & Prompts"]:::data
        History["Historical Runs Archive (.history)"]:::data
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        PG["Prompt Generator"]:::process
        CEF["create_empty_files.py"]:::process
        RAND["random.py"]:::process
        DERAND["derandom.py"]:::process
        LLM["LLM API Client"]:::process
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        Results["Results Store (by domain)"]:::storage
    end

    %% External Services Layer
    subgraph "External Services"
        direction TB
        OpenAI["OpenAI API"]:::external
        Sheets["Google Sheets (Aggregator)"]:::external
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        direction TB
        Visual["Static Charts (visual/)"]:::viz
    end

    %% Data Flows
    PDDL -->|"generate .prompt"| PG
    History -->|"use history"| PG
    PG -->|"call API"| LLM
    LLM -->|"save outputs"| Results
    Results -->|"aggregate metrics"| Sheets
    Sheets -->|"produce charts"| Visual
    Results -->|"generate local charts"| Visual

    %% Click Events
    click PDDL "https://github.com/vita-group/o1-planning/tree/main/results/"
    click History "https://github.com/vita-group/o1-planning/tree/main/results/*/.history/"
    click CEF "https://github.com/vita-group/o1-planning/blob/main/scripts/create_empty_files.py"
    click RAND "https://github.com/vita-group/o1-planning/blob/main/results/randomized_tyreworld_test2/random.py"
    click DERAND "https://github.com/vita-group/o1-planning/blob/main/results/randomized_tyreworld_test2/derandom.py"
    click Results "https://github.com/vita-group/o1-planning/tree/main/results/"
    click Sheets "https://github.com/vita-group/o1-planning/blob/main/README.md"
    click Visual "https://github.com/vita-group/o1-planning/blob/main/visual/radar_chart.pdf"

    %% Styles
    classDef data fill:#e0ffe0,stroke:#0a0,stroke-width:1px
    classDef process fill:#e0e8ff,stroke:#00f,stroke-width:1px
    classDef external fill:#ffe8cc,stroke:#f80,stroke-width:1px
    classDef storage fill:#fff5cc,stroke:#cc0,stroke-width:1px
    classDef viz fill:#f0e0ff,stroke:#80f,stroke-width:1px
```
