How batsresearch/planetarium works
Project Type & Purpose
• This is an evaluation & dataset tool focused on converting and comparing PDDL problem descriptions.
• The key goal is to transform PDDL (and its corresponding natural language descriptions) into graph representations, “fully specify” these graphs (by adding necessary propositions), and then perform isomorphism checks to assess translation correctness.
• In addition to core evaluation, the project provides scripts for dataset generation (dataset_generator.py), finetuning (finetune.py), and planning (llm_planner.py).
Main Components & Their Placement
a. Input & Dataset Module:
• Dataset generator (dataset_generator.py and associated config files from the "experiment_cfgs" directory).
• Integration with external dataset repositories (linking to Hugging Face datasets as outlined in the README).
b. Core Processing / Evaluation Pipeline:
• PDDL Input: Mark a component for receiving two PDDL problem descriptions.
• Graph Construction:
• A node for “Graph Representation”, which is implemented by modules like planetarium/graph.py and planetarium/reduced_graph.py.
• That the transformation from PDDL to graph is central.
• Fully Specify Function:
• This module enriches the graph by adding “trivial” propositions. It is domain-specific. Reference planetarium/oracles (blocksworld.py, gripper.py, etc.) which implement the function for each domain.
• Graph Comparison / Isomorphism:
• A component representing the “Graph Isomorphism” checker. This is the core evaluation step that compares the fully specified graphs.
• The planetarium/metric.py module might also be used here to measure the similarity or correctness.
c. External Modules & Integrations:
• Integrate external planning & validation tools:
• Additional Components:
• “llm_planner.py” and “finetune.py” indicate there’s support for LLM-based planning as well as training/evaluation components. Represent these as optional modules that feed into or extend the evaluation pipeline.
Relationships & Data Flow
• Next, direct the flow into the "Fully Specified" module (using domain-specific oracle implementations) which enriches or adjusts the graph.
• Then, illustrate that the enhanced graph is sent to the Graph Isomorphism checker.
• Finally, the outcome (evaluation metrics) is provided by the evaluate.py script/module.
• Separately, a branch can show the “dataset generation” process and its configuration feed (via dataset_generator.py and experiment_cfgs files).
Architectural Patterns & Design Principles
• Modular Design: The project is clearly organized into modules grouped by purpose (graph building, evaluation, domain-specific functions, dataset management).
• Separation of Concerns: There is a clear segregation between graph transformation, evaluation, and data management.
• Extensibility: The “EXTENDING.md” and modular oracle design illustrate an architecture that invites the addition of new domains.
• Integration with External Systems: Show how the system relies on external planners (Fast-Downward and VAL) for parts of its functionality.
Technologies & Libraries
• Python is used as the primary language, with dependency management handled via Poetry and pyproject.toml.
• External planners: Fast-Downward (via Apptainer) and VAL.
• The project also utilizes data from the Hugging Face hub, evident from the README badges and dataset links.