# internrobotics/genmanip architecture

> Architecture diagram of the GitHub repository internrobotics/genmanip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internrobotics/genmanip
- Repository: https://github.com/internrobotics/genmanip

## Overview

1. **Overall shape**
   This repository is a robotics simulation and dataset-generation platform built around **NVIDIA Isaac Sim**. The codebase is organized as a reusable Python package (`genmanip`) plus a large set of **configuration-driven assets** and **task definitions** under `configs/` and `assets/`.

2. **Primary architectural boundary: code vs. content**
   The repository cleanly separates executable logic from simulation content:
   - `genmanip/` contains the runtime framework, planners, evaluators, skills, robot abstractions, and utilities.
   - `assets/` contains robot descriptions, meshes, object annotations, and URDF/SRDF/mesh artifacts.
   - `configs/` defines cameras, robot setups, task suites, and benchmark splits.  
   This means most behavior is assembled by config rather than hardcoded in the core.

3. **Core framework**
   `genmanip/core/` provides the abstract interfaces and orchestration layer:
   - `scene/` for scene composition and scene config
   - `robot/` for robot-agnostic interfaces
   - `skill/` for task primitives
   - `metrics/` for evaluation logic
   - `evaluator/` for running simulations and collecting results  
   The core is where the “unified API” described in the README is implemented.

4. **Robot abstraction layer**
   The framework supports multiple embodiments through a common robot interface:
   - single-arm and dual-arm manipulation in `genmanip/core/robot/`
   - concrete robot extensions in `genmanip/extensions/robots/default/` for Franka/Panda, Robotiq, Aloha Split, and Lift2  
   Robot-specific kinematics, TCP definitions, and collision/visual descriptions live in the assets and config layer, while the runtime logic stays generic.

5. **Scene and asset management**
   Scene assembly relies on structured loaders and object metadata:
   - `genmanip/utils/loader/` handles scene loading, asset search, hardcoded rules, preload rules, and domain randomization
   - `genmanip/utils/annotation/` and `assets/objects/*.json|pickle` provide object pools and semantic/articulation annotations
   - `genmanip/utils/usd_utils/` contains low-level USD editing/export utilities for camera, physics, collisions, materials, semantics, and transforms  
   This is the main boundary between abstract task definition and the concrete Isaac Sim/USD scene graph.

6. **Skill and action layer**
   Task behavior is composed from reusable skills:
   - default skills in `genmanip/extensions/skills/default/` include `move_to`, `pick`, `pick_and_place`, and `custom_motion`
   - core skill base types live in `genmanip/core/skill/`
   These skills sit between high-level task rules and low-level robot motion planning.

7. **Planning and motion execution**
   Motion planning is delegated to external planners and wrapped by utilities:
   - `genmanip/utils/planner/curobo/` for **cuRobo**
   - `genmanip/utils/planner/mplib/` for alternative planning utilities
   - `genmanip/utils/anygrasp/` for grasp proposal support  
   In the architecture, planning is a separate service-like layer consumed by skills and demo/generation workflows rather than embedded in scene logic.

8. **Generation pipeline**
   Data generation is centered in `genmanip/demogen/`:
   - `workflow/demogen.py` and `workflow/render.py` orchestrate generation and rendering
   - `random_place/` implements placement strategies for objects in scenes
   - `recoder/` captures planning traces and rendered outputs  
   This subsystem turns a scene/task config into synthetic demonstrations, likely producing training data compatible with the repo’s stated LeRobot GR00t conventions.

9. **Evaluation and benchmarking**
   Benchmarks are executed through the evaluator stack:
   - `genmanip/core/evaluator/` contains environment management, worker pools, eval server, progress/logging, and exceptions
   - `genmanip/extensions/metrics/` adds benchmark-specific metrics for the “exciting benchmark” and default relationship/orientation checks
   - top-level `ray_eval_server.py` suggests distributed evaluation using **Ray**
   The evaluation layer is decoupled from generation so tasks can be scored across many parallel Isaac Sim instances.

10. **Benchmark suites and task catalogs**
   `configs/tasks/` is the main catalog of benchmark definitions. It includes:
   - `ebench/` with generalist, mobile manipulation, and tabletop benchmark splits
   - `iros_challenge_refined_*` suites
   - `scaling_up/` large-object and long-horizon scenarios
   - demo/minimal configs for quick runs  
   Each task family is represented by multiple split files and per-task YAMLs, which the runtime loads to instantiate benchmark scenarios.

11. **Camera and robot configuration**
   `configs/cameras/` and `configs/robots/` define reusable sensor and robot presets. These configs are important because they determine:
   - view geometry and fixed camera setups
   - robot TCP frames and end-effector setups
   - variant-specific simulation boxes or alignment settings  
   They are the main inputs that make the same task portable across embodiments and scene layouts.

12. **Tooling, demos, and operational scripts**
   The repo includes several operational entrypoints outside the package:
   - `demogen.py`, `render.py`, and `ray_eval_server.py` are top-level runners
   - `demos/` contains example scene export, replay, and visualization flows
   - `standalone_tools/` contains asset download, conversion, cleanup, verification, and analysis utilities  
   These scripts support the content pipeline: ingest assets, convert them to simulation-ready formats, run generation/evaluation, and analyze results.

13. **Data and interchange formats**
   The repository uses a mix of robotics and simulation formats:
   - URDF/SRDF for robot structure and planning semantics
   - USD-related utilities and exported scenes for Isaac Sim
   - STL/GLB/DAE meshes for collision and visualization
   - YAML/JSON for configs and benchmark manifests
   - pickle/JSON annotations for object metadata and articulation info  
   The diagram should reflect these as external content stores feeding the runtime rather than as code modules.

14. **Key runtime dependencies that shape architecture**
   The main external systems are **Isaac Sim**, **cuRobo**, and **Ray**. Isaac Sim is the simulation substrate, cuRobo is the motion-planning backend, and Ray enables parallel evaluation at scale. The repository’s architecture is therefore “config-driven robotics simulation + distributed generation/evaluation,” not a conventional application server.

Main entry points: [`genmanip/demogen`](https://github.com/internrobotics/genmanip/tree/main/genmanip/demogen) (Demogen), [`genmanip/core/evaluator`](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/evaluator) (Evaluator core), [`genmanip/core/scene`](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/scene) (Scene core), [`genmanip/core/skill`](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/skill) (Skill core)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_genmanip_core_scene["Scene core<br/>scene orchestration"]
  node_genmanip_core_robot["Robot core<br/>robot abstraction"]
  node_genmanip_core_skill["Skill core<br/>skill primitives"]
  node_genmanip_core_metrics["Metrics core<br/>evaluation metrics"]
  node_genmanip_core_evaluator["Evaluator core<br/>eval orchestration"]
  node_genmanip_extensions_robots["Robot presets<br/>embodiment adapters"]
  node_genmanip_extensions_skills["Default skills<br/>task skills"]
  node_genmanip_extensions_metrics["Benchmark metrics<br/>domain metrics"]
  node_genmanip_utils_loader["Scene loader<br/>asset loading"]
  node_genmanip_utils_usd["USD utils<br/>scene editing"]
  node_genmanip_utils_planner["Planners<br/>motion planning"]
  node_genmanip_demogen["Demogen<br/>data generation"]
end

subgraph group_content["Content"]
  node_configs_tasks["Task configs<br/>benchmark manifests"]
  node_configs_cameras["Camera configs<br/>sensor presets"]
  node_configs_robots["Robot configs<br/>robot presets"]
  node_assets_objects[("Object assets<br/>annotations")]
  node_assets_robots[("Robot assets<br/>robot descriptions")]
end

subgraph group_ops["Operations"]
  node_ray_eval_server(("Ray eval<br/>distributed eval<br/>[ray_eval_server.py]"))
  node_demogen_py(("Gen runner<br/>entrypoint<br/>[demogen.py]"))
  node_render_py(("Render runner<br/>entrypoint<br/>[render.py]"))
  node_standalone_tools["Tools<br/>asset pipeline"]
end

subgraph group_external["External systems"]
  node_isaac_sim[("Isaac Sim<br/>simulation backend")]
  node_curobo[("cuRobo<br/>planner backend")]
  node_ray[("Ray<br/>parallel runtime")]
end

node_configs_tasks -->|"define scenes"| node_genmanip_core_scene
node_configs_cameras -->|"preset cameras"| node_genmanip_utils_loader
node_configs_robots -->|"preset robots"| node_genmanip_core_robot
node_assets_objects -->|"object metadata"| node_genmanip_utils_loader
node_assets_robots -->|"robot models"| node_genmanip_extensions_robots
node_genmanip_utils_loader -->|"assemble scene"| node_genmanip_core_scene
node_genmanip_utils_usd -->|"edit USD"| node_genmanip_core_scene
node_genmanip_core_scene -->|"instantiate world"| node_isaac_sim
node_genmanip_extensions_robots -->|"implement robots"| node_genmanip_core_robot
node_genmanip_core_robot -->|"request plans"| node_genmanip_utils_planner
node_genmanip_utils_planner -->|"plan motions"| node_curobo
node_genmanip_extensions_skills -->|"implement skills"| node_genmanip_core_skill
node_genmanip_core_skill -->|"execute motion"| node_genmanip_utils_planner
node_genmanip_core_skill -->|"command robot"| node_genmanip_core_robot
node_genmanip_demogen -->|"sample scenes"| node_genmanip_core_scene
node_genmanip_demogen -->|"run skills"| node_genmanip_core_skill
node_genmanip_demogen -->|"render outputs"| node_render_py
node_genmanip_demogen -->|"place objects"| node_genmanip_utils_loader
node_genmanip_core_evaluator -->|"run episodes"| node_isaac_sim
node_genmanip_core_evaluator -->|"parallelize"| node_ray
node_ray_eval_server -->|"serve eval"| node_genmanip_core_evaluator
node_render_py -->|"render workflow"| node_genmanip_demogen
node_demogen_py -->|"launch gen"| node_genmanip_demogen
node_standalone_tools -->|"process assets"| node_assets_objects
node_standalone_tools -->|"convert robots"| node_assets_robots
node_genmanip_extensions_metrics -->|"extend metrics"| node_genmanip_core_metrics
node_genmanip_core_evaluator -->|"score runs"| node_genmanip_core_metrics
node_configs_tasks -->|"drive generation"| node_genmanip_demogen
node_configs_tasks -->|"drive eval"| node_genmanip_core_evaluator

click node_genmanip_core_scene "https://github.com/internrobotics/genmanip/tree/main/genmanip/core/scene"
click node_genmanip_core_robot "https://github.com/internrobotics/genmanip/tree/main/genmanip/core/robot"
click node_genmanip_core_skill "https://github.com/internrobotics/genmanip/tree/main/genmanip/core/skill"
click node_genmanip_core_metrics "https://github.com/internrobotics/genmanip/tree/main/genmanip/core/metrics"
click node_genmanip_core_evaluator "https://github.com/internrobotics/genmanip/tree/main/genmanip/core/evaluator"
click node_genmanip_extensions_robots "https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/robots/default"
click node_genmanip_extensions_skills "https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/skills/default"
click node_genmanip_extensions_metrics "https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/metrics"
click node_genmanip_utils_loader "https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/loader"
click node_genmanip_utils_usd "https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/usd_utils"
click node_genmanip_utils_planner "https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/planner"
click node_genmanip_demogen "https://github.com/internrobotics/genmanip/tree/main/genmanip/demogen"
click node_configs_tasks "https://github.com/internrobotics/genmanip/tree/main/configs/tasks"
click node_configs_cameras "https://github.com/internrobotics/genmanip/tree/main/configs/cameras"
click node_configs_robots "https://github.com/internrobotics/genmanip/tree/main/configs/robots"
click node_assets_objects "https://github.com/internrobotics/genmanip/tree/main/assets/objects"
click node_assets_robots "https://github.com/internrobotics/genmanip/tree/main/assets/robots"
click node_ray_eval_server "https://github.com/internrobotics/genmanip/blob/main/ray_eval_server.py"
click node_demogen_py "https://github.com/internrobotics/genmanip/blob/main/demogen.py"
click node_render_py "https://github.com/internrobotics/genmanip/blob/main/render.py"
click node_standalone_tools "https://github.com/internrobotics/genmanip/tree/main/standalone_tools"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_genmanip_core_scene,node_genmanip_core_robot,node_genmanip_core_skill,node_genmanip_core_metrics,node_genmanip_core_evaluator,node_genmanip_extensions_robots,node_genmanip_extensions_skills,node_genmanip_extensions_metrics,node_genmanip_utils_loader,node_genmanip_utils_usd,node_genmanip_utils_planner,node_genmanip_demogen toneBlue
class node_configs_tasks,node_configs_cameras,node_configs_robots,node_assets_objects,node_assets_robots toneAmber
class node_ray_eval_server,node_demogen_py,node_render_py,node_standalone_tools toneMint
class node_isaac_sim,node_curobo,node_ray toneRose
```

## Components

### Runtime

Python package that orchestrates scene setup, robots, skills, planning, evaluation, and generation.

- [Scene core](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/scene): scene orchestration · Abstract scene composition and scene configuration layer. · `genmanip/core/scene`
- [Robot core](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/robot): robot abstraction · Robot-agnostic interfaces for single-arm and dual-arm manipulation. · `genmanip/core/robot`
- [Skill core](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/skill): skill primitives · Base task-action interfaces used to compose behaviors. · `genmanip/core/skill`
- [Metrics core](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/metrics): evaluation metrics · Shared metric interfaces and metric management. · `genmanip/core/metrics`
- [Evaluator core](https://github.com/internrobotics/genmanip/tree/main/genmanip/core/evaluator): eval orchestration · Environment management, workers, progress, and result collection for simulations. · `genmanip/core/evaluator`
- [Robot presets](https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/robots/default): embodiment adapters · Concrete robot extensions for Panda/Franka, Robotiq, Aloha Split, and Lift2. · `genmanip/extensions/robots/default`
- [Default skills](https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/skills/default): task skills · Reusable behavior implementations such as move, pick, pick-and-place, and custom motion. · `genmanip/extensions/skills/default`
- [Benchmark metrics](https://github.com/internrobotics/genmanip/tree/main/genmanip/extensions/metrics): domain metrics · Task-specific and benchmark-specific metric implementations. · `genmanip/extensions/metrics`
- [Scene loader](https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/loader): asset loading · Asset search, preload rules, hardcoded rules, and domain randomization. · `genmanip/utils/loader`
- [USD utils](https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/usd_utils): scene editing · Low-level USD camera, physics, collision, material, semantic, and transform helpers. · `genmanip/utils/usd_utils`
- [Planners](https://github.com/internrobotics/genmanip/tree/main/genmanip/utils/planner): motion planning · Wrappers for cuRobo, mplib, and grasp proposal support. · `genmanip/utils/planner`
- [Demogen](https://github.com/internrobotics/genmanip/tree/main/genmanip/demogen): data generation · Workflow for synthetic demonstrations, rendering, random placement, and recording. · `genmanip/demogen`

### Content

Config-driven task, camera, robot, and asset definitions consumed by the runtime.

- [Task configs](https://github.com/internrobotics/genmanip/tree/main/configs/tasks): benchmark manifests · Benchmark suites, task splits, and scenario YAML/JSON definitions. · `configs/tasks`
- [Camera configs](https://github.com/internrobotics/genmanip/tree/main/configs/cameras): sensor presets · Reusable fixed and benchmark-specific camera configurations. · `configs/cameras`
- [Robot configs](https://github.com/internrobotics/genmanip/tree/main/configs/robots): robot presets · Robot TCP and planning presets used to instantiate embodiments. · `configs/robots`
- [Object assets](https://github.com/internrobotics/genmanip/tree/main/assets/objects): annotations · Object pools, articulation metadata, and semantic annotations. · `assets/objects`
- [Robot assets](https://github.com/internrobotics/genmanip/tree/main/assets/robots): robot descriptions · URDF/SRDF and mesh artifacts for robot embodiments and sensors. · `assets/robots`

### Operations

Entry points, demos, and tooling for generation, rendering, evaluation, and asset processing.

- [Ray eval](https://github.com/internrobotics/genmanip/blob/main/ray_eval_server.py): distributed eval · Top-level distributed evaluation server entry point. · `ray_eval_server.py`
- [Gen runner](https://github.com/internrobotics/genmanip/blob/main/demogen.py): entrypoint · Top-level demonstration generation entry point. · `demogen.py`
- [Render runner](https://github.com/internrobotics/genmanip/blob/main/render.py): entrypoint · Top-level rendering entry point. · `render.py`
- [Tools](https://github.com/internrobotics/genmanip/tree/main/standalone_tools): asset pipeline · Asset download, conversion, cleanup, validation, and analysis utilities. · `standalone_tools`

### External systems

Simulation, planning, and distributed execution backends.

- **Isaac Sim**: simulation backend · Primary robotics simulation substrate.
- **cuRobo**: planner backend · External motion planning backend.
- **Ray**: parallel runtime · Distributed execution backend for evaluation at scale.

## Connections

- Task configs → Scene core: define scenes · Task manifests drive scene composition and scenario setup.
- Camera configs → Scene loader: preset cameras · Camera configs are loaded into scene assembly and rendering.
- Robot configs → Robot core: preset robots · Robot configs instantiate embodiment parameters and TCPs.
- Object assets → Scene loader: object metadata · Object annotations and pools feed scene/object selection.
- Robot assets → Robot presets: robot models · URDF/SRDF and meshes support robot-specific adapters.
- Scene loader → Scene core: assemble scene · Loader resolves assets and applies placement/randomization rules.
- USD utils → Scene core: edit USD · USD utilities modify cameras, physics, collisions, and semantics.
- Scene core → Isaac Sim: instantiate world · Scene definitions are realized inside Isaac Sim.
- Robot presets → Robot core: implement robots · Concrete robot packages extend the abstract robot interface.
- Robot core → Planners: request plans · Robot execution delegates motion planning to planner wrappers.
- Planners → cuRobo: plan motions · cuRobo is the primary planning backend.
- Default skills → Skill core: implement skills · Default skills provide reusable manipulation primitives.
- Skill core → Planners: execute motion · Skills invoke planning utilities to produce robot actions.
- Skill core → Robot core: command robot · Skills target the abstract robot interface.
- Demogen → Scene core: sample scenes · Generation workflows create randomized scene/task instances.
- Demogen → Skill core: run skills · Demogen executes task primitives during demonstration synthesis.
- Demogen → Render runner: render outputs · Generation workflow can hand off to rendering.
- Demogen → Scene loader: place objects · Random placement and scene graph placement rely on loaders.
- Evaluator core → Isaac Sim: run episodes · Evaluator manages simulation instances and workers.
- Evaluator core → Ray: parallelize · Distributed evaluation is scheduled through Ray.
- Ray eval → Evaluator core: serve eval · Server entry point exposes evaluator orchestration.
- Render runner → Demogen: render workflow · Rendering runner invokes the generation/render pipeline.
- Gen runner → Demogen: launch gen · Top-level generation script starts demogen workflows.
- Tools → Object assets: process assets · Operational tools preprocess, validate, and analyze content assets.
- Tools → Robot assets: convert robots · Tools convert and clean robot descriptions and meshes.
- Benchmark metrics → Metrics core: extend metrics · Benchmark metrics build on shared metric interfaces.
- Evaluator core → Metrics core: score runs · Evaluator computes results through the metrics layer.
- Task configs → Demogen: drive generation · Task suites configure which scenes and episodes are generated.
- Task configs → Evaluator core: drive eval · Benchmark splits and task configs parameterize evaluation runs.