# wiscohumanoids/manipulation_sim architecture

> Architecture diagram of the GitHub repository wiscohumanoids/manipulation_sim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wiscohumanoids/manipulation_sim
- Repository: https://github.com/wiscohumanoids/manipulation_sim

## Overview

manipulation_sim is a simulated robotics-learning sandbox. Its main walkthrough collects scripted Fetch demonstrations, converts them to a LeRobot dataset, trains an ACT policy, then evaluates it in the environment. Separate branches provide a Jacobian-IK pick-and-place baseline, Diffusion Policy, PPO reach/pick-place training, and an interactive policy launcher. Edges reflect README workflow statements or shown imports/calls; unsampled approach internals are represented only where the README documents their role.

Main entry points: [`approaches/imitation_act/collect_demos.py`](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/collect_demos.py) (Demo collection), [`play.py`](https://github.com/wiscohumanoids/manipulation_sim/blob/main/play.py) (Interactive launcher), [`envs/arm_pick_place.py`](https://github.com/wiscohumanoids/manipulation_sim/blob/main/envs/arm_pick_place.py) (Fetch tasks), [`approaches/scripted_ik/pick_place_scripted.py`](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/scripted_ik/pick_place_scripted.py) (IK pick-place)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_simulation["Simulation"]
  node_fetch["Fetch tasks<br/>[arm_pick_place.py]"]
  node_reach["Reach task<br/>[arm_reach.py]"]
end

subgraph group_classical["Scripted control"]
  node_ik_demo["IK pick-place"]
  node_ik_solver["Jacobian IK<br/>[ik_solver.py]"]
end

subgraph group_imitation["Imitation learning"]
  node_expert["Scripted expert<br/>[scripted_expert.py]"]
  node_collect["Demo collection<br/>[collect_demos.py]"]
  node_episodes["Raw episodes<br/>[recording.py]"]
  node_format["Dataset formatting<br/>[format_dataset.py]"]
  node_dataset[("LeRobot dataset<br/>[format_dataset.py]")]
  node_act_train["ACT training<br/>[train_act.py]"]
  node_diff_train["Diffusion training<br/>[train_diffusion.py]"]
  node_act_eval["ACT evaluation<br/>[evaluate.py]"]
  node_diff_eval["Diffusion evaluation<br/>[evaluate.py]"]
  node_checkpoint["Policy checkpoint<br/>[train_act.py]"]
end

subgraph group_rl["Reinforcement learning"]
  node_ppo["PPO training<br/>[train_rl.py]"]
end

subgraph group_runtime["Interactive runtime"]
  node_video["Video output<br/>[recording.py]"]
  node_launcher["Interactive launcher<br/>[play.py]"]
  node_paths["Output paths<br/>[repo_paths.py]"]
  node_viz["MuJoCo rendering<br/>[viz.py]"]
end

node_user(("Learner"))
node_hub[("Hugging Face Hub")]

node_user -->|"run baseline"| node_ik_demo
node_ik_demo -->|"solves with"| node_ik_solver
node_user -->|"collect demos"| node_collect
node_collect -->|"steps task"| node_fetch
node_collect -->|"requests actions"| node_expert
node_collect -->|"saves episodes"| node_episodes
node_format -->|"loads episodes"| node_episodes
node_format -->|"creates dataset"| node_dataset
node_user -->|"format demos"| node_format
node_format -.->|"optionally pushes"| node_hub
node_user -->|"train policy"| node_act_train
node_act_train -->|"trains from"| node_dataset
node_act_train -->|"saves checkpoint"| node_checkpoint
node_user -->|"choose diffusion"| node_diff_train
node_diff_train -.->|"trains from"| node_dataset
node_diff_train -.->|"saves policy"| node_checkpoint
node_user -->|"evaluate policy"| node_act_eval
node_act_eval -->|"runs policy in"| node_fetch
node_act_eval -->|"loads checkpoint"| node_checkpoint
node_act_eval -->|"records video"| node_video
node_diff_eval -->|"runs policy in"| node_fetch
node_diff_eval -->|"records video"| node_video
node_user -->|"train PPO"| node_ppo
node_ppo -->|"selects pick-place"| node_fetch
node_ppo -->|"selects reach"| node_reach
node_user -->|"launch interactively"| node_launcher
node_launcher -->|"steps environment"| node_fetch
node_launcher -->|"records oracle demos"| node_expert
node_launcher -->|"discovers policy"| node_checkpoint
node_ik_demo -->|"renders with"| node_viz
node_ik_demo -->|"writes recording"| node_video
node_launcher -.->|"renders frames"| node_viz
node_collect -->|"uses output paths"| node_paths
node_act_train -->|"uses output paths"| node_paths

click node_fetch "https://github.com/wiscohumanoids/manipulation_sim/blob/main/envs/arm_pick_place.py"
click node_reach "https://github.com/wiscohumanoids/manipulation_sim/blob/main/envs/arm_reach.py"
click node_ik_demo "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/scripted_ik/pick_place_scripted.py"
click node_ik_solver "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/scripted_ik/ik_solver.py"
click node_expert "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/scripted_expert.py"
click node_collect "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/collect_demos.py"
click node_episodes "https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/recording.py"
click node_format "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/format_dataset.py"
click node_dataset "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/format_dataset.py"
click node_act_train "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/train_act.py"
click node_diff_train "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_diffusion/train_diffusion.py"
click node_act_eval "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/evaluate.py"
click node_diff_eval "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_diffusion/evaluate.py"
click node_checkpoint "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/train_act.py"
click node_ppo "https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/rl_grasp/train_rl.py"
click node_video "https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/recording.py"
click node_launcher "https://github.com/wiscohumanoids/manipulation_sim/blob/main/play.py"
click node_paths "https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/repo_paths.py"
click node_viz "https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/viz.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetch,node_reach toneBlue
class node_ik_demo,node_ik_solver,node_hub toneAmber
class node_expert,node_collect,node_episodes,node_format,node_dataset,node_act_train,node_diff_train,node_act_eval,node_diff_eval,node_checkpoint toneMint
class node_ppo toneRose
class node_video,node_launcher,node_paths,node_viz,node_user toneIndigo
```

## Components

### Simulation

- [Fetch tasks](https://github.com/wiscohumanoids/manipulation_sim/blob/main/envs/arm_pick_place.py): `envs/arm_pick_place.py`
- [Reach task](https://github.com/wiscohumanoids/manipulation_sim/blob/main/envs/arm_reach.py): `envs/arm_reach.py`

### Scripted control

- [IK pick-place](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/scripted_ik/pick_place_scripted.py): `approaches/scripted_ik/pick_place_scripted.py`
- [Jacobian IK](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/scripted_ik/ik_solver.py): `approaches/scripted_ik/ik_solver.py`

### Imitation learning

- [Scripted expert](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/scripted_expert.py): `approaches/imitation_act/scripted_expert.py`
- [Demo collection](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/collect_demos.py): `approaches/imitation_act/collect_demos.py`
- [Raw episodes](https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/recording.py): `utils/recording.py`
- [Dataset formatting](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/format_dataset.py): `approaches/imitation_act/format_dataset.py`
- [LeRobot dataset](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/format_dataset.py): `approaches/imitation_act/format_dataset.py`
- [ACT training](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/train_act.py): `approaches/imitation_act/train_act.py`
- [Diffusion training](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_diffusion/train_diffusion.py): `approaches/imitation_diffusion/train_diffusion.py`
- [ACT evaluation](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/evaluate.py): `approaches/imitation_act/evaluate.py`
- [Diffusion evaluation](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_diffusion/evaluate.py): `approaches/imitation_diffusion/evaluate.py`
- [Policy checkpoint](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/imitation_act/train_act.py): `approaches/imitation_act/train_act.py`

### Reinforcement learning

- [PPO training](https://github.com/wiscohumanoids/manipulation_sim/blob/main/approaches/rl_grasp/train_rl.py): `approaches/rl_grasp/train_rl.py`

### Interactive runtime

- [Video output](https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/recording.py): `utils/recording.py`
- [Interactive launcher](https://github.com/wiscohumanoids/manipulation_sim/blob/main/play.py): `play.py`
- [Output paths](https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/repo_paths.py): `utils/repo_paths.py`
- [MuJoCo rendering](https://github.com/wiscohumanoids/manipulation_sim/blob/main/utils/viz.py): `utils/viz.py`

### Other components

- **Learner**
- **Hugging Face Hub**

## Connections

- Learner → IK pick-place: run baseline (evidence: `README.md`)
- IK pick-place → Jacobian IK: solves with (evidence: `approaches/scripted_ik/pick_place_scripted.py`)
- Learner → Demo collection: collect demos (evidence: `README.md`)
- Demo collection → Fetch tasks: steps task (evidence: `approaches/imitation_act/collect_demos.py`)
- Demo collection → Scripted expert: requests actions (evidence: `approaches/imitation_act/collect_demos.py`)
- Demo collection → Raw episodes: saves episodes (evidence: `approaches/imitation_act/collect_demos.py`)
- Dataset formatting → Raw episodes: loads episodes (evidence: `approaches/imitation_act/format_dataset.py`)
- Dataset formatting → LeRobot dataset: creates dataset (evidence: `approaches/imitation_act/format_dataset.py`)
- Learner → Dataset formatting: format demos (evidence: `README.md`)
- Dataset formatting → Hugging Face Hub: optionally pushes (evidence: `README.md`)
- Learner → ACT training: train policy (evidence: `README.md`)
- ACT training → LeRobot dataset: trains from (evidence: `README.md`)
- ACT training → Policy checkpoint: saves checkpoint (evidence: `README.md`)
- Learner → Diffusion training: choose diffusion
- Diffusion training → LeRobot dataset: trains from (evidence: `approaches/imitation_act/format_dataset.py`)
- Diffusion training → Policy checkpoint: saves policy (evidence: `approaches/imitation_diffusion/train_diffusion.py`)
- Learner → ACT evaluation: evaluate policy (evidence: `README.md`)
- ACT evaluation → Fetch tasks: runs policy in (evidence: `approaches/imitation_act/evaluate.py`)
- ACT evaluation → Policy checkpoint: loads checkpoint (evidence: `README.md`)
- ACT evaluation → Video output: records video (evidence: `approaches/imitation_act/evaluate.py`)
- Diffusion evaluation → Fetch tasks: runs policy in (evidence: `approaches/imitation_diffusion/evaluate.py`)
- Diffusion evaluation → Video output: records video (evidence: `approaches/imitation_diffusion/evaluate.py`)
- Learner → PPO training: train PPO (evidence: `approaches/rl_grasp/train_rl.py`)
- PPO training → Fetch tasks: selects pick-place (evidence: `approaches/rl_grasp/train_rl.py`)
- PPO training → Reach task: selects reach (evidence: `approaches/rl_grasp/train_rl.py`)
- Learner → Interactive launcher: launch interactively (evidence: `play.py`)
- Interactive launcher → Fetch tasks: steps environment (evidence: `play.py`)
- Interactive launcher → Scripted expert: records oracle demos (evidence: `play.py`)
- Interactive launcher → Policy checkpoint: discovers policy (evidence: `play.py`)
- IK pick-place → MuJoCo rendering: renders with (evidence: `approaches/scripted_ik/pick_place_scripted.py`)
- IK pick-place → Video output: writes recording (evidence: `approaches/scripted_ik/pick_place_scripted.py`)
- Interactive launcher → MuJoCo rendering: renders frames (evidence: `play.py`)
- Demo collection → Output paths: uses output paths (evidence: `approaches/imitation_act/collect_demos.py`)
- ACT training → Output paths: uses output paths (evidence: `approaches/imitation_act/train_act.py`)
