How internrobotics/genmanip works
1. Overall shape
This repository is a robotics simulation and dataset-generation platform built around NVIDIA Isaac Sim. The codebase is organized as a reusable Python package (genmanip) plus a large set of configuration-driven assets and task definitions under configs/ and assets/.
2. Primary architectural boundary: code vs. content
The repository cleanly separates executable logic from simulation content:
- genmanip/ contains the runtime framework, planners, evaluators, skills, robot abstractions, and utilities.
- assets/ contains robot descriptions, meshes, object annotations, and URDF/SRDF/mesh artifacts.
- configs/ defines cameras, robot setups, task suites, and benchmark splits.
This means most behavior is assembled by config rather than hardcoded in the core.
3. Core framework
genmanip/core/ provides the abstract interfaces and orchestration layer:
- scene/ for scene composition and scene config
- robot/ for robot-agnostic interfaces
- skill/ for task primitives
- metrics/ for evaluation logic
- evaluator/ for running simulations and collecting results
The core is where the “unified API” described in the README is implemented.
4. Robot abstraction layer
The framework supports multiple embodiments through a common robot interface:
- single-arm and dual-arm manipulation in genmanip/core/robot/
- concrete robot extensions in genmanip/extensions/robots/default/ for Franka/Panda, Robotiq, Aloha Split, and Lift2
Robot-specific kinematics, TCP definitions, and collision/visual descriptions live in the assets and config layer, while the runtime logic stays generic.
5. Scene and asset management
Scene assembly relies on structured loaders and object metadata:
- genmanip/utils/loader/ handles scene loading, asset search, hardcoded rules, preload rules, and domain randomization
- genmanip/utils/annotation/ and assets/objects/*.json|pickle provide object pools and semantic/articulation annotations
- genmanip/utils/usd_utils/ contains low-level USD editing/export utilities for camera, physics, collisions, materials, semantics, and transforms
This is the main boundary between abstract task definition and the concrete Isaac Sim/USD scene graph.
6. Skill and action layer
Task behavior is composed from reusable skills:
- default skills in genmanip/extensions/skills/default/ include move_to, pick, pick_and_place, and custom_motion
- core skill base types live in genmanip/core/skill/
These skills sit between high-level task rules and low-level robot motion planning.
7. Planning and motion execution
Motion planning is delegated to external planners and wrapped by utilities:
- genmanip/utils/planner/curobo/ for cuRobo
- genmanip/utils/planner/mplib/ for alternative planning utilities
- genmanip/utils/anygrasp/ for grasp proposal support
In the architecture, planning is a separate service-like layer consumed by skills and demo/generation workflows rather than embedded in scene logic.
8. Generation pipeline
Data generation is centered in genmanip/demogen/:
- workflow/demogen.py and workflow/render.py orchestrate generation and rendering
- random_place/ implements placement strategies for objects in scenes
- recoder/ captures planning traces and rendered outputs
This subsystem turns a scene/task config into synthetic demonstrations, likely producing training data compatible with the repo’s stated LeRobot GR00t conventions.
9. Evaluation and benchmarking
Benchmarks are executed through the evaluator stack:
- genmanip/core/evaluator/ contains environment management, worker pools, eval server, progress/logging, and exceptions
- genmanip/extensions/metrics/ adds benchmark-specific metrics for the “exciting benchmark” and default relationship/orientation checks
- top-level ray_eval_server.py suggests distributed evaluation using Ray
The evaluation layer is decoupled from generation so tasks can be scored across many parallel Isaac Sim instances.
10. Benchmark suites and task catalogs
configs/tasks/ is the main catalog of benchmark definitions. It includes:
- ebench/ with generalist, mobile manipulation, and tabletop benchmark splits
- iros_challenge_refined_* suites
- scaling_up/ large-object and long-horizon scenarios
- demo/minimal configs for quick runs
Each task family is represented by multiple split files and per-task YAMLs, which the runtime loads to instantiate benchmark scenarios.
11. Camera and robot configuration
configs/cameras/ and configs/robots/ define reusable sensor and robot presets. These configs are important because they determine:
- view geometry and fixed camera setups
- robot TCP frames and end-effector setups
- variant-specific simulation boxes or alignment settings
They are the main inputs that make the same task portable across embodiments and scene layouts.
12. Tooling, demos, and operational scripts
The repo includes several operational entrypoints outside the package:
- demogen.py, render.py, and ray_eval_server.py are top-level runners
- demos/ contains example scene export, replay, and visualization flows
- standalone_tools/ contains asset download, conversion, cleanup, verification, and analysis utilities
These scripts support the content pipeline: ingest assets, convert them to simulation-ready formats, run generation/evaluation, and analyze results.
13. Data and interchange formats
The repository uses a mix of robotics and simulation formats:
- URDF/SRDF for robot structure and planning semantics
- USD-related utilities and exported scenes for Isaac Sim
- STL/GLB/DAE meshes for collision and visualization
- YAML/JSON for configs and benchmark manifests
- pickle/JSON annotations for object metadata and articulation info
The diagram should reflect these as external content stores feeding the runtime rather than as code modules.
14. Key runtime dependencies that shape architecture
The main external systems are Isaac Sim, cuRobo, and Ray. Isaac Sim is the simulation substrate, cuRobo is the motion-planning backend, and Ray enables parallel evaluation at scale. The repository’s architecture is therefore “config-driven robotics simulation + distributed generation/evaluation,” not a conventional application server.
Main entry points: demogen Demogen · evaluator Evaluator core · scene Scene core · skill Skill core
Components 24
Runtime
Python package that orchestrates scene setup, robots, skills, planning, evaluation, and generation.
- Scene core · Abstract scene composition and scene configuration layer.
genmanip/core/scene - Robot core · Robot-agnostic interfaces for single-arm and dual-arm manipulation.
genmanip/core/robot - Skill core · Base task-action interfaces used to compose behaviors.
genmanip/core/skill - Metrics core · Shared metric interfaces and metric management.
genmanip/core/metrics - Evaluator core · Environment management, workers, progress, and result collection for simulations.
genmanip/core/evaluator - Robot presets · Concrete robot extensions for Panda/Franka, Robotiq, Aloha Split, and Lift2.
genmanip/extensions/robots/default - Default skills · Reusable behavior implementations such as move, pick, pick-and-place, and custom motion.
genmanip/extensions/skills/default - Benchmark metrics · Task-specific and benchmark-specific metric implementations.
genmanip/extensions/metrics - Scene loader · Asset search, preload rules, hardcoded rules, and domain randomization.
genmanip/utils/loader - USD utils · Low-level USD camera, physics, collision, material, semantic, and transform helpers.
genmanip/utils/usd_utils - Planners · Wrappers for cuRobo, mplib, and grasp proposal support.
genmanip/utils/planner - Demogen · Workflow for synthetic demonstrations, rendering, random placement, and recording.
genmanip/demogen
Content
Config-driven task, camera, robot, and asset definitions consumed by the runtime.
- Task configs · Benchmark suites, task splits, and scenario YAML/JSON definitions.
configs/tasks - Camera configs · Reusable fixed and benchmark-specific camera configurations.
configs/cameras - Robot configs · Robot TCP and planning presets used to instantiate embodiments.
configs/robots - Object assets · Object pools, articulation metadata, and semantic annotations.
assets/objects - Robot assets · URDF/SRDF and mesh artifacts for robot embodiments and sensors.
assets/robots
Operations
Entry points, demos, and tooling for generation, rendering, evaluation, and asset processing.
- Ray eval · Top-level distributed evaluation server entry point.
ray_eval_server.py - Gen runner · Top-level demonstration generation entry point.
demogen.py - Render runner · Top-level rendering entry point.
render.py - Tools · Asset download, conversion, cleanup, validation, and analysis utilities.
standalone_tools
External systems
Simulation, planning, and distributed execution backends.
- Isaac Sim · Primary robotics simulation substrate.
- cuRobo · External motion planning backend.
- Ray · Distributed execution backend for evaluation at scale.
Connections 29
- Task configs → Scene core: define scenes · Task manifests drive scene composition and scenario setup.
- Camera configs → Scene loader: preset cameras · Camera configs are loaded into scene assembly and rendering.
- Robot configs → Robot core: preset robots · Robot configs instantiate embodiment parameters and TCPs.
- Object assets → Scene loader: object metadata · Object annotations and pools feed scene/object selection.
- Robot assets → Robot presets: robot models · URDF/SRDF and meshes support robot-specific adapters.
- Scene loader → Scene core: assemble scene · Loader resolves assets and applies placement/randomization rules.
- USD utils → Scene core: edit USD · USD utilities modify cameras, physics, collisions, and semantics.
- Scene core → Isaac Sim: instantiate world · Scene definitions are realized inside Isaac Sim.
- Robot presets → Robot core: implement robots · Concrete robot packages extend the abstract robot interface.
- Robot core → Planners: request plans · Robot execution delegates motion planning to planner wrappers.
- Planners → cuRobo: plan motions · cuRobo is the primary planning backend.
- Default skills → Skill core: implement skills · Default skills provide reusable manipulation primitives.
- Skill core → Planners: execute motion · Skills invoke planning utilities to produce robot actions.
- Skill core → Robot core: command robot · Skills target the abstract robot interface.
- Demogen → Scene core: sample scenes · Generation workflows create randomized scene/task instances.
- Demogen → Skill core: run skills · Demogen executes task primitives during demonstration synthesis.
- Demogen → Render runner: render outputs · Generation workflow can hand off to rendering.
- Demogen → Scene loader: place objects · Random placement and scene graph placement rely on loaders.
- Evaluator core → Isaac Sim: run episodes · Evaluator manages simulation instances and workers.
- Evaluator core → Ray: parallelize · Distributed evaluation is scheduled through Ray.
- Ray eval → Evaluator core: serve eval · Server entry point exposes evaluator orchestration.
- Render runner → Demogen: render workflow · Rendering runner invokes the generation/render pipeline.
- Gen runner → Demogen: launch gen · Top-level generation script starts demogen workflows.
- Tools → Object assets: process assets · Operational tools preprocess, validate, and analyze content assets.
- Tools → Robot assets: convert robots · Tools convert and clean robot descriptions and meshes.
- Benchmark metrics → Metrics core: extend metrics · Benchmark metrics build on shared metric interfaces.
- Evaluator core → Metrics core: score runs · Evaluator computes results through the metrics layer.
- Task configs → Demogen: drive generation · Task suites configure which scenes and episodes are generated.
- Task configs → Evaluator core: drive eval · Benchmark splits and task configs parameterize evaluation runs.