How haosulab/maniskill works
Main Components
• Simulation & Rendering Engine: Highlight the integration with SAPIEN and Vulkan for rendering. Emphasize the GPU-parallelized simulation aspects (both state-based and heterogeneous environments).
• Environment & Scene Management: Include the “mani_skill/envs” and “mani_skill/utils/scene_builder” modules. This area manages different types of simulation environments, ranging from basic (minimal_template, empty_env) to complex tasks (tabletop, mobile manipulation, dexterity, digital twins).
• Agents, Controllers & Robots: Show components like “mani_skill/agents” which hosts base agents, controllers, and robot models (data in “mani_skill/agents/robots” and “mani_skill/assets/robots”). These represent how behaviors are defined and how various robot embodiments (e.g., panda, humanoid, so100) are structured.
• Task and Data Collection Modules: Represent “mani_skill/envs/tasks” and “mani_skill/utils/scene_builder” for task construction, data collection strategies, and integration of object meshes and assets.
• Learning Baselines & Examples: Include the “examples” directory that demonstrates various learning approaches (e.g., PPO, SAC, TD-MPC2, diffusion policy) as well as demonstration scripts.
• Utility & Build Infrastructure: Group smaller but equally important modules such as “mani_skill/utils”, “scripts”, and “tests” for asset building, logging, configuration management, and testing.
• Documentation & CI/CD: Outline the roles of the “docs” directory and GitHub workflows (in “.github/workflows”) for continuous integration and publishing.
Relationships and Data Flow
• For instance, “Agents” interact with “Environments” by sending control commands and receiving sensor observations.
• How learning algorithms (located in examples and baselines) interface with the simulation via APIs provided in modules like “mani_skill/agents” and “mani_skill/envs”.
Architectural Patterns and Design Principles
• Modular and Object-Oriented Design: Each major module (agents, envs, utils) serves as a self-contained component with clear interfaces for extension.
• GPU-Parallelization: Indicate that simulation and rendering are optimized for high throughput using GPU computation—this is a central design innovation.
• Separation of Concerns: Separate simulation logic, rendering, task configuration, asset loading, and training examples. This separation is evident across directories (e.g., “mani_skill/agents” vs. “mani_skill/envs” vs. “mani_skill/utils”).
• Extensibility: Note the simple task building API and configurable components (e.g., controllers and scene builders) that allow research users to extend or replace different parts.
Relevant Technologies and Dependencies
• SAPIEN and Vulkan are foundational to simulation and rendering.
• The use of PyTorch is implied (installation steps show “pip install torch”), which is critical for the learning (reinforcement and imitation learning) modules.
• Docker support (files in the “docker” directory) indicates containerized deployment/testing.
• CI/CD is managed through GitHub Actions workflows.