How ethz-asl/vgn works
Main System Components
• Data Generation & Dataset Construction:
• Components include “scripts/generate_data.py” for simulating grasp trials with pybullet (physics simulation), “scripts/construct_dataset.py” for converting raw simulation output into voxel grids/grasp targets, and associated notebooks in "notebooks/" for cleaning and visualizing data.
• Neural Network Training:
• Involve “scripts/train_vgn.py” and the core network implementation files in “src/vgn/” such as “networks.py”, “dataset.py”, and possibly “baselines.py”.
• TensorBoard logging (data/runs) should be noted as a feedback channel.
• Simulation and Evaluation:
• The system simulates grasping experiments using “scripts/sim_grasp.py” along with ROS visualization configurations from “config/sim.rviz”, “config/gpd/ros_eigen_params.cfg” and the “launch” files.
• There is also a clutter removal experiment in “src/vgn/experiments/clutter_removal.py” and the “notebooks/clutter_removal.ipynb” for evaluation.
• Robot Grasp Execution:
• The integration with hardware is shown via “scripts/panda_grasp.py”, “launch/panda_grasp.launch”, and additional ROS integration in “src/vgn/utils/ros_utils.py” and “config/panda.yaml”.
• The experimental setup involves a Franka Emika Panda robot with a Realsense sensor and the MoveIt configuration.
• Configuration & Integration:
• The project’s configuration files (YAMLs and RViz configurations in “config/”) and ROS launch files (in “launch/”) define environment, visualization, and simulation parameters.
• Build and package management are supported by CMakeLists.txt, package.xml, setup.py, and requirements.txt.
Out Relationships and Interactions
• Data Flow:
• Dataset Construction then processes the raw data into a format that the neural network can use.
• Next, Network Training using the processed datasets creates the trained model (e.g., vgn_conv.pth).
• Inference & Simulation:
• The trained model is then used in simulated grasping (sim_grasp.py) and is visualized using ROS RViz configurations.
• Robot Interaction:
• For real-world experiments, the trained model is deployed via scripts like “panda_grasp.py”, interfacing with ROS nodes over a distributed system (robot drivers on one machine and grasp scripts on another).
• External Dependencies:
• The system uses ROS for visualization and robot interfacing, pybullet for simulation, and optionally OpenMPI for distributed data generation.
Architectural Patterns
• Modularity:
• The system is organized into distinct components (data generation, training, simulation, and robot execution). Each component’s logic resides in separate directories (scripts, notebooks, src/vgn).
• Layered Approach:
• There is a clear separation between the back-end (data handling, neural network training, simulation) and the interface layer (ROS, visualization, robot control).
• Integration with ROS:
• ROS nodes, launch files, and RViz configurations indicate a service-based or event-driven communication for real-time monitoring and hardware interfacing.