How flagopen/action-sketcher works
Action-Sketcher is a visual-reasoning and robot-action research system for long-horizon manipulation. Its documented inference workflow starts with a LIBERO evaluation run, uses a PI0 policy to produce reasoning or actions, and can render visual sketches; the method describes these sketches as an intermediate plan for action generation. Separate training code and dataset modules support reasoning-only and joint action-language training. The graph preserves the documented LIBERO environment and lifelong-learning capabilities as well. Source excerpts are limited, so undocumented runtime wiring is omitted or marked unverified.
Main entry points: run_libero_example.py LIBERO Run · inference.py Inference API · modeling.py PI0 Policy · train_action_language.py Joint Training
Components 17
Inference and Sketching
- LIBERO Run
run_libero_example.py - Inference API
inference.py - Sketch Rendering
draw_vp.py
PI0 Policy
- PI0 Policy
pi0/modeling.py - PaliGemma Expert
pi0/paligemma_with_expert.py - PI0 Utilities
pi0/utils.py - LIBERO Policy Config
lerobot/common/policies/pi0/configuration_pi0_libero.py
Training and Data
- Reasoning Training
scripts/train_language.py - Joint Training
scripts/train_action_language.py - LIBERO Dataset
robotics_dataset/dataset_complete_1_libero.py - LeRobot Dataset
lerobot/common/datasets/lerobot_dataset.py - Lifelong Dataset
libero/lifelong/datasets.py
LIBERO Benchmark
- LIBERO Environments
libero/libero/envs/bddl_base_domain.py - Task Definitions
libero/libero/envs/problems - Lifelong Learning
libero/lifelong/main.py - Benchmark Evaluation
libero/lifelong/metric.py
Other components
- Evaluator
Connections 16
- Evaluator → LIBERO Run: starts evaluation
README.md - LIBERO Run → Inference API: calls
run_libero_example.py - LIBERO Run → Sketch Rendering: uses
run_libero_example.py - Inference API → PI0 Policy: runs policy
- PI0 Policy → PaliGemma Expert: uses backbone
pi0/modeling.py - PI0 Policy → PI0 Utilities: uses utilities
pi0/modeling.py - Inference API → LIBERO Policy Config: loads configuration
inference.py - Evaluator → Reasoning Training: trains reasoning
README.md - Evaluator → Joint Training: trains jointly
README.md - Reasoning Training → LIBERO Dataset: uses training data
- Joint Training → LIBERO Dataset: uses training data
- Lifelong Learning → Lifelong Dataset: loads datasets
libero/lifelong/main.py - Lifelong Learning → Benchmark Evaluation: evaluates tasks
libero/lifelong/main.py - Benchmark Evaluation → LIBERO Environments: evaluates in
libero/lifelong/metric.py - Task Definitions → LIBERO Environments: defines tasks
libero/libero/envs/problems/libero_living_room_tabletop_manipulation.py - Joint Training → LeRobot Dataset: uses dataset