How nepfaff/scalable-real2sim works
Main Components
• Pipeline Entrypoints:
• Data Collection (run_data_collection.py)
• Asset Generation (run_asset_generation.py)
• Data Processing Module: (scalable_real2sim/data_processing); includes routines for handling images, masks, COLMAP data, neural network inputs, etc.
• Submodules & External Components:
• BundleSDF (object tracking and reconstruction)
• Frosting (surface reconstruction using Gaussian Splatting)
• Neuralangelo (neural surface reconstruction)
• nerfstudio (NeRF-based processing)
• robot_payload_id (robot and payload identification)
• Segmentation: (scalable_real2sim/segmentation) for detect_object and segment_moving_object_data routines.
• Additional Scripts: (scripts directory) for post-processing and metric computations, such as geometric metrics and mask subtraction.
Relationships and Data Flow
• Data Collection Phase:
• Data is acquired via run_data_collection.py, which interacts with the robot setup (possibly invoking pickplace_data_collection and robot_payload_id) and writes data to a shared output directory.
• This phase involves capturing sensor or joint data as well as object and scene information.
• Pre-Processing & Processing:
• Data from the collection phase is then sent into data_processing routines, where segmentation (object detection, masks) and transformation (colmap, neuralangelo dataset preparation) are applied.
• Asset Generation Phase:
• run_asset_generation.py consumes the processed data and, possibly in a distributed or asynchronous manner, sends it through several reconstruction pipelines.
• Each submodule (BundleSDF, Frosting, neuralangelo, nerfstudio) works as a distinct process (often with its own virtual environment/setup) to perform high-end object reconstruction and texture mapping.
• External Tools & Dependencies:
Architectural Patterns and Technologies
• Modularity: The system is highly modular (different directories hinting at a microservices-like approach, even within a single repository) with clear separation between data collection and asset generation.
• Distributed Execution Possibility: The separation of the asset generation script from the data collection script implies that these can run in different environments (e.g., local robot versus cloud).
• Robotics and System Identification: Including the robot_payload_id module and associated scripts for trajectory optimization and joint data collection is crucial to understanding how the physical robot setup integrates with the digital pipeline.