How skumra/robotic-grasping works
Project Type and Purpose
• This is a full-stack research prototype for antipodal robotic grasping. It combines a deep learning training pipeline (GR-ConvNet models), data preprocessing utilities, real-time inference, and hardware interface components for camera calibration and robot execution.
File Structure Analysis
• Top-level scripts:
• train_network.py, evaluate.py – model training and evaluation
• run_*.py – task runners (calibration, grasp generation, offline, realtime)
• inference/ – model definitions (grconvnet, grconvnet2–4), grasp generator, post-processing
• hardware/ – camera calibration, device wrappers, camera interface
• utils/ – data loaders (Cornell, Jacquard), dataset processing (depth generation, evaluation), visualization, timing
• trained-models/ – model artifact store
• requirements.txt, config (_config.yml), cleanup scripts
README Insights
• Core architecture: Generative Residual Convolutional Neural Network for predicting antipodal grasps
• Supports two datasets (Cornell, Jacquard) with preprocessing steps
• Real-time inference using pyrealsense2 camera feed or offline evaluation
• ROS integration via separate Baxter PnP repository
Main Components
a. Data Layer
• Raw datasets (Cornell, Jacquard)
• Preprocessing modules (utils.dataset_processing: generate depths, evaluation)
b. Training Pipeline
• train_network.py
• inference.models (GR-ConvNet variants)
• TensorBoardX for logging
• Model artifacts (trained-models/)
c. Inference Pipeline
• run_grasp_generator.py, run_offline.py, run_realtime.py
• inference.grasp_generator (loads model, runs forward pass)
• inference.post_process (non-max suppression, grasp filtering)
d. Hardware Interface
• hardware.camera (pyrealsense2 wrapper)
• hardware.calibrate_camera (intrinsic/extrinsic calibration)
• hardware.device (robot or simulator interface)
e. Visualization & Utilities
• utils.visualisation (gridshow, plotting)
• utils.timeit, utils.data (camera_data, grasp_data)
Component Relationships and Data Flow
• Data Layer → Training Pipeline: raw images → depth generation → batch loader → training script → saved model
• Trained Model → Inference Pipeline: model file → grasp_generator → grasp proposals
• Inference Pipeline ↔ Hardware Interface: camera feed → inference → device commands
• Visualization modules tap into both training (TensorBoard plots) and inference (gridshow) for diagnostics
Architectural Patterns and Design Principles
• Modular separation: data utilities, training, inference, hardware drivers
• Pipeline orchestration: clear data → model → hardware flow
• Extensibility: swapping model variants (grconvnet2,3,4) via common interface
• Script-based entry points for tasks (run_*.py)
Technologies and Frameworks
• Python 3.6+, PyTorch, TorchVision, TensorBoardX
• OpenCV, scikit-image, imageio for preprocessing
• pyrealsense2 for camera interface
• Optional ROS integration for robot control