# flagopen/action-sketcher architecture

> Architecture diagram of the GitHub repository flagopen/action-sketcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flagopen/action-sketcher
- Repository: https://github.com/flagopen/action-sketcher

## Overview

Action-Sketcher is a visual-reasoning and robot-action research system for long-horizon manipulation. Its documented inference workflow starts with a LIBERO evaluation run, uses a PI0 policy to produce reasoning or actions, and can render visual sketches; the method describes these sketches as an intermediate plan for action generation. Separate training code and dataset modules support reasoning-only and joint action-language training. The graph preserves the documented LIBERO environment and lifelong-learning capabilities as well. Source excerpts are limited, so undocumented runtime wiring is omitted or marked unverified.

Main entry points: [`run_libero_example.py`](https://github.com/flagopen/action-sketcher/blob/main/run_libero_example.py) (LIBERO Run), [`inference.py`](https://github.com/flagopen/action-sketcher/blob/main/inference.py) (Inference API), [`pi0/modeling.py`](https://github.com/flagopen/action-sketcher/blob/main/pi0/modeling.py) (PI0 Policy), [`scripts/train_action_language.py`](https://github.com/flagopen/action-sketcher/blob/main/scripts/train_action_language.py) (Joint Training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference and Sketching"]
  node_evaluation_run["LIBERO Run"]
  node_inference_api["Inference API<br/>[inference.py]"]
  node_sketch_render["Sketch Rendering<br/>[draw_vp.py]"]
end

subgraph group_policy["PI0 Policy"]
  node_pi0_policy["PI0 Policy<br/>[modeling.py]"]
  node_paligemma_expert["PaliGemma Expert"]
  node_pi0_utils["PI0 Utilities<br/>[utils.py]"]
  node_policy_config["LIBERO Policy Config"]
end

subgraph group_training["Training and Data"]
  node_train_reason["Reasoning Training<br/>[train_language.py]"]
  node_train_joint["Joint Training"]
  node_robot_data[("LIBERO Dataset")]
  node_lerobot_data[("LeRobot Dataset<br/>[lerobot_dataset.py]")]
  node_lifelong_data[("Lifelong Dataset<br/>[datasets.py]")]
end

subgraph group_libero["LIBERO Benchmark"]
  node_libero_env["LIBERO Environments"]
  node_task_definitions["Task Definitions"]
  node_lifelong_learning["Lifelong Learning<br/>[main.py]"]
  node_lifelong_eval["Benchmark Evaluation<br/>[metric.py]"]
end

node_user(("Evaluator"))

node_user -->|"starts evaluation"| node_evaluation_run
node_evaluation_run -->|"calls"| node_inference_api
node_evaluation_run -->|"uses"| node_sketch_render
node_inference_api -.->|"runs policy"| node_pi0_policy
node_pi0_policy -->|"uses backbone"| node_paligemma_expert
node_pi0_policy -->|"uses utilities"| node_pi0_utils
node_inference_api -->|"loads configuration"| node_policy_config
node_user -->|"trains reasoning"| node_train_reason
node_user -->|"trains jointly"| node_train_joint
node_train_reason -.->|"uses training data"| node_robot_data
node_train_joint -.->|"uses training data"| node_robot_data
node_lifelong_learning -->|"loads datasets"| node_lifelong_data
node_lifelong_learning -->|"evaluates tasks"| node_lifelong_eval
node_lifelong_eval -->|"evaluates in"| node_libero_env
node_task_definitions -->|"defines tasks"| node_libero_env
node_train_joint -.->|"uses dataset"| node_lerobot_data

click node_evaluation_run "https://github.com/flagopen/action-sketcher/blob/main/run_libero_example.py"
click node_inference_api "https://github.com/flagopen/action-sketcher/blob/main/inference.py"
click node_sketch_render "https://github.com/flagopen/action-sketcher/blob/main/draw_vp.py"
click node_pi0_policy "https://github.com/flagopen/action-sketcher/blob/main/pi0/modeling.py"
click node_paligemma_expert "https://github.com/flagopen/action-sketcher/blob/main/pi0/paligemma_with_expert.py"
click node_pi0_utils "https://github.com/flagopen/action-sketcher/blob/main/pi0/utils.py"
click node_policy_config "https://github.com/flagopen/action-sketcher/blob/main/lerobot/common/policies/pi0/configuration_pi0_libero.py"
click node_train_reason "https://github.com/flagopen/action-sketcher/blob/main/scripts/train_language.py"
click node_train_joint "https://github.com/flagopen/action-sketcher/blob/main/scripts/train_action_language.py"
click node_robot_data "https://github.com/flagopen/action-sketcher/blob/main/robotics_dataset/dataset_complete_1_libero.py"
click node_lerobot_data "https://github.com/flagopen/action-sketcher/blob/main/lerobot/common/datasets/lerobot_dataset.py"
click node_lifelong_data "https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/datasets.py"
click node_libero_env "https://github.com/flagopen/action-sketcher/blob/main/libero/libero/envs/bddl_base_domain.py"
click node_task_definitions "https://github.com/flagopen/action-sketcher/tree/main/libero/libero/envs/problems"
click node_lifelong_learning "https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/main.py"
click node_lifelong_eval "https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/metric.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_evaluation_run,node_inference_api,node_sketch_render toneBlue
class node_pi0_policy,node_paligemma_expert,node_pi0_utils,node_policy_config toneAmber
class node_train_reason,node_train_joint,node_robot_data,node_lerobot_data,node_lifelong_data toneMint
class node_libero_env,node_task_definitions,node_lifelong_learning,node_lifelong_eval toneRose
class node_user toneIndigo
```

## Components

### Inference and Sketching

- [LIBERO Run](https://github.com/flagopen/action-sketcher/blob/main/run_libero_example.py): `run_libero_example.py`
- [Inference API](https://github.com/flagopen/action-sketcher/blob/main/inference.py): `inference.py`
- [Sketch Rendering](https://github.com/flagopen/action-sketcher/blob/main/draw_vp.py): `draw_vp.py`

### PI0 Policy

- [PI0 Policy](https://github.com/flagopen/action-sketcher/blob/main/pi0/modeling.py): `pi0/modeling.py`
- [PaliGemma Expert](https://github.com/flagopen/action-sketcher/blob/main/pi0/paligemma_with_expert.py): `pi0/paligemma_with_expert.py`
- [PI0 Utilities](https://github.com/flagopen/action-sketcher/blob/main/pi0/utils.py): `pi0/utils.py`
- [LIBERO Policy Config](https://github.com/flagopen/action-sketcher/blob/main/lerobot/common/policies/pi0/configuration_pi0_libero.py): `lerobot/common/policies/pi0/configuration_pi0_libero.py`

### Training and Data

- [Reasoning Training](https://github.com/flagopen/action-sketcher/blob/main/scripts/train_language.py): `scripts/train_language.py`
- [Joint Training](https://github.com/flagopen/action-sketcher/blob/main/scripts/train_action_language.py): `scripts/train_action_language.py`
- [LIBERO Dataset](https://github.com/flagopen/action-sketcher/blob/main/robotics_dataset/dataset_complete_1_libero.py): `robotics_dataset/dataset_complete_1_libero.py`
- [LeRobot Dataset](https://github.com/flagopen/action-sketcher/blob/main/lerobot/common/datasets/lerobot_dataset.py): `lerobot/common/datasets/lerobot_dataset.py`
- [Lifelong Dataset](https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/datasets.py): `libero/lifelong/datasets.py`

### LIBERO Benchmark

- [LIBERO Environments](https://github.com/flagopen/action-sketcher/blob/main/libero/libero/envs/bddl_base_domain.py): `libero/libero/envs/bddl_base_domain.py`
- [Task Definitions](https://github.com/flagopen/action-sketcher/tree/main/libero/libero/envs/problems): `libero/libero/envs/problems`
- [Lifelong Learning](https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/main.py): `libero/lifelong/main.py`
- [Benchmark Evaluation](https://github.com/flagopen/action-sketcher/blob/main/libero/lifelong/metric.py): `libero/lifelong/metric.py`

### Other components

- **Evaluator**

## Connections

- Evaluator → LIBERO Run: starts evaluation (evidence: `README.md`)
- LIBERO Run → Inference API: calls (evidence: `run_libero_example.py`)
- LIBERO Run → Sketch Rendering: uses (evidence: `run_libero_example.py`)
- Inference API → PI0 Policy: runs policy
- PI0 Policy → PaliGemma Expert: uses backbone (evidence: `pi0/modeling.py`)
- PI0 Policy → PI0 Utilities: uses utilities (evidence: `pi0/modeling.py`)
- Inference API → LIBERO Policy Config: loads configuration (evidence: `inference.py`)
- Evaluator → Reasoning Training: trains reasoning (evidence: `README.md`)
- Evaluator → Joint Training: trains jointly (evidence: `README.md`)
- Reasoning Training → LIBERO Dataset: uses training data
- Joint Training → LIBERO Dataset: uses training data
- Lifelong Learning → Lifelong Dataset: loads datasets (evidence: `libero/lifelong/main.py`)
- Lifelong Learning → Benchmark Evaluation: evaluates tasks (evidence: `libero/lifelong/main.py`)
- Benchmark Evaluation → LIBERO Environments: evaluates in (evidence: `libero/lifelong/metric.py`)
- Task Definitions → LIBERO Environments: defines tasks (evidence: `libero/libero/envs/problems/libero_living_room_tabletop_manipulation.py`)
- Joint Training → LeRobot Dataset: uses dataset
