How nvlabs/mimicgen works
Project type and purpose
• This is an open-source Python toolkit (MimicGen) for generating large-scale robot learning datasets via simulated environments.
• It provides:
• A CLI/API layer (scripts) to launch data-generation runs.
• A core data-generation engine configurable by JSON configs.
• Wrappers around the robosuite simulation environments.
• Assets/models for simulated robots and objects.
• Utility functions and experiment templates.
File structure
• Top-level packages under mimicgen/:
• configs: configuration definitions for tasks and environments
• datagen: core data-generation pipeline (data_generator, selection_strategy, waypoint, etc.)
• env_interfaces: abstract base and robosuite wrappers
• envs/robosuite: concrete env definitions for each task (coffee, pick_place, etc.)
• exps/templates: JSON templates for experiment configurations
• models: XML/STL assets and object definitions for simulation
• scripts: CLI entry points for generating datasets, configs, videos, etc.
• utils: file, pose, config, robomimic utilities
• Support files: setup.py, requirements, add_license_header.py
• Documentation under docs/ with Sphinx source.
README
• Confirms purpose: “data generation system for scalable robot learning.”
• Mentions 48,000+ demonstrations, configurable runs, extant datasets on external storage (Hugging Face).
1. CLI/Scripts Layer
• Entry points: generate_dataset.py, download_datasets.py, etc.
2. Config Module
• Loads JSON/py configs (mimicgen.configs + templates)
3. Data-Generation Engine
• Core pipeline: DataGenerator, SelectionStrategy, Waypoint
4. Environment Interface
• Base interface → RobosuiteInterface
5. Simulation Environments
• Task-specific env classes (coffee, hammer_cleanup, etc.)
6. Models & Assets
• XML, STL, OBJ in mimicgen/models for object meshes and scene definitions
7. Utils
• File I/O, pose conversions, robomimic helpers
8. Data Output
• HDF5 logs, video recordings, JSON metadata
9. External Services
• Dataset hosting (Hugging Face, Google Drive)
• CLI/Scripts → reads Config Module → instantiates Data-Generation Engine
• Engine → calls Environment Interface → spawns Simulation Env and loads Models & Assets
• Simulation Env → executes episodes, returns observations → Engine processes with Waypoint & Strategy
• Engine → writes outputs via Utils to Data Output storage
• Scripts optionally fetch existing datasets from External Services
c. Architectural patterns & principles
• Modularity: clear separation between config, engine, env interfaces, and assets
• Strategy pattern: selection_strategy, waypoint sampling abstracted
• Plugin-style envs: new tasks extend base interface
• Config-driven: JSON templates + Python config definitions
• Reusability: utils and common code support multiple scripts and pipelines
d. Key technologies & libraries
• Python 3.x, robosuite simulator
• HDF5 (h5py) for dataset storage
• Sphinx for docs
• Command-line scripts
Detail
• How JSON templates in exps/templates/robosuite feed into the config loader
• Outputs (HDF5, video, JSON logs) and optionally the dataset upload process to Hugging Face
• Non-obvious dependencies (e.g., scripts use config_utils to assemble configs before passing to the engine)