How pku-epic/graspnerf works
Main Components and Their Roles
• Data Generation & Simulation:
• Components: The “data_generator” directory and simulation scripts (e.g., run_simgrasp.sh, run_pile_rand.sh in the data_generator folder) indicate that synthetic data generation, scene description modification, and pile randomization are key. This feeds training data and simulation assets.
• Integration: These scripts generate scene descriptors and use assets (where textures from ImageNet are used) required by both simulation (blender and pybullet) and training pipelines.
• Grasp Detection (GD) Pipeline:
• Component: The “src/gd” directory contains modules for detection, baseline methods, dataset management, grasp detection logic, experiments (including clutter removal), perception, I/O, neural networks integrations, and visualization.
• Function: This part of the system handles the core grasp detection logic using multiview inputs, potentially interfacing with the neural rendering components to leverage the NeRF for scene representation.
• Neural Radiance Field (NR) Pipeline:
• Component: The “src/nr” directory houses network architecture code (including distinct networks like aggregate_net, dist_decoder, IBRNet, MVSNet modules, neus, etc.), training routines, loss functions, and utility scripts.
• Function: Implements the neural rendering / scene representation construction, enabling the zero-shot, sparse RGB-based NeRF construction.
• Rendering (RD) Module:
• Component: The “src/rd” directory includes scripts like modify_material.py, render.py, and render_utils.py.
• Function: Handles rendering tasks likely tailored for simulation visualization, material modification, and interfacing with the graphics pipeline. It may work closely with the simulation engine to help display or validate grasp results.
• Supporting Modules & Utilities:
• Components: “scripts” directory and various utility files in both “src/gd/utils” and “src/nr/utils”.
• Function: Provide orchestration code (sim_grasp.py to run simulations, stat_expresult.py for statistical results), dataset utilities, transformation functions (including ROS utils and Panda control), and generic helper functions that underpin both training and simulation pipelines.
Relationships and Interactions
• Training: The NR and GD modules are trained end-to-end; therefore, illustrate how the neural rendering (NR) front integrates with grasp detection (GD) to optimize the scene representation for grasp planning.
• Simulation Loop: Post-training, simulation scripts (using blender and pybullet) and the rendering module (RD) work together to validate grasp detection in realistic scenes.
Important Patterns and Design Principles
• Modularity: The repository is partitioned into distinct modules (GD, NR, RD, and Data Generation) which interact in a clearly defined manner.
• End-to-End Learning: Emphasize that training jointly optimizes both the NeRF (NR) and grasp detection (GD) components, a design principle that is central to this work.
• Simulation and Real-World Transfer: Highlight the separation between offline synthetic data generation, the training pipeline, and the online simulation/validation pipeline. This separation is key to ensuring the model generalizes to real-world grasping tasks.
Depicting Technologies and Tools
• Framework: PyTorch (for neural network training) is the backbone.
• Simulation Tools: Blender and PyBullet are used for simulation and generating rendering assets.
• Data Assets: External datasets (e.g., ImageNet Val, provided checkpoints, synthetic scene descriptors) support both the training and simulation.
• Scripts/Configuration: Shell scripts (like train.sh, run_simgrasp.sh) orchestrate the high-level workflows.