How h-freax/thinkgrasp works
Main Components
• Input & Data Sources:
– Sensor Input Module: This includes camera inputs (RGB and depth) and text goal descriptions used in real-world tests (as seen in the API payload for realarm.py).
– Simulation Data: Test cases (testing_cases, heavy_unseen, assets of objects/URDF models) and external datasets from assets.
• Vision-Language Pipeline:
– Vision Processing: The “VLP” directory contains modules for GLIP, segment-anything, and even VLPart. These modules leverage maskrcnn_benchmark and segmentation neural networks to extract object and part features.
– Language Processing: Language backbones are found in the glip directory and in “langsam” (which contains language and actor modules). BERT models and tokenizers are used (as shown by the bert_model.py files) to process language inputs.
• Grasp Inference Module:
– GraspNet Models: In the “models” folder you have two variations (FGC_graspnet and graspnet) that include dataset handling, training scripts (train.py), inference modules, and optimized KNN/PointNet2 implementations.
– Grasp Detector: The file “grasp_detetor.py” also indicates part of the inference/control pipeline.
• Environment & Simulation:
– Simulation Engine: Files like simulation_main.py and environment_sim.py build a simulated environment (using PyBullet) to test negotiations with obstacles and clutter.
– ROS/URDF Assets: The assets folder provides URDFs and mesh files that are used for simulating robot hardware and the environment.
• Real-World Interface:
– Flask API Server: The realarm.py script runs a Flask application that exposes an API endpoint (/grasp_pose) for controlling or triggering the grasp process in real-world scenarios.
• Utility & Infrastructure Modules:
– Core Utilities: Files like constants.py, logger.py, engine.py, and utils.py support configuration, logging, and miscellaneous functions.
– Model Build & Setup: Setup scripts (setup.py, requirements.txt, langsam.txt, vlp_requirements.txt) handle package and dependency management for various subcomponents.
• External Dependencies:
– Hardware & Accelerator Support: PyTorch, CUDA (note CUDA installation instructions and GPU recommendations), PyBullet for simulation, and WandB for experiment tracking.
– Compilation & Custom Ops: The multiple build directories (e.g., for knn and pointnet2 inside models) indicate custom CUDA/C++ kernels integration.
Representing Relationships and Data Flow
a. Real-World Input (from sensors) that flows to the Real-World Interface (Flask server “realarm.py”).
b. Simulation Input (using pre-saved test cases and assets) that flows to the Simulation Engine (“simulation_main.py”).
– From these, create a central processing pipeline where sensor data or images enters the Vision-Language Pipeline. In this zone:
• Visual Data is processed by pre-trained vision networks (GLIP, segment-anything from VLP/glip/ and VLP/segment-anything) to extract features, semantic segmentation, and object proposals.
• Language Input (goal/task text) is encoded using BERT/language backbones (files in glip and langsam).
• The two modalities are fused (via modules in “maskrcnn_benchmark” and “vlpart”) to generate context-aware representations.
– The fused representation is sent into the Grasp Inference Module, which comprises of:
• The FGC_graspnet and graspnet models (including their dataset, training, and inference pipelines).
• Custom KNN and PointNet2 operations for geometric reasoning.
– The output (grasp pose predictions) is then processed by a control/decision block that routes commands to:
• The Simulator (for testing and visualization in simulation_main.py).
• The real robot controller (invoked via the Flask API for realarm.py) to execute physical grasps.
– Add supporting utility and logging interactions that monitor each stage (often connected to WandB for experiment tracking and logging events).
Architectural Patterns & Design Principles
– Modular & Layered Architecture: Each major functionality (vision-language processing, grasp inference, simulation/real-world execution) is isolated into its own directory and module.
– Separation of Concerns: The project cleanly separates simulation versus real-world pipelines and decouples core inference logic from environment integration.
– Hardware-Accelerated Processing: Emphasize use of CUDA-enabled modules, custom C++ extensions, and multi-GPU support (noting the two GTX 3090 recommendation).
– Extensibility: The code structure (e.g., “setup.py”, multiple requirements files) supports rapid prototyping and addition of new modules.
Instructing the Principal Software Engineer
– Map out the Vision-Language Pipeline with its two branches (visual and language) that converge in a fusion module.
– Next, illustrate the Grasp Inference Module block showing the integration of FGC_graspnet, graspnet, and supporting KNN/PointNet2 custom kernels.
– Finally, list external dependencies (PyTorch, CUDA, PyBullet, Flask) and utility components (logger, configuration, experiment tracking via WandB) as ancillary elements that interface with the main processing pipeline.