How wmtlab/graingrasp works
Project type and purpose
• GrainGrasp is a research codebase for “Dexterous Grasp Generation with Fine-grained Contact Guidance.”
• It provides both training and inference pipelines for generating hand grasps on 3D objects using point-cloud neural networks (PointCVAE, SupNet) and a differentiable hand model (MANO).
• Key subcomponents: data processing, model training, inference/optimization, and visualization.
File structure
Top-level directories and files reveal the major subsystems:
• “PointCVAE/” and “SupNet/”: two neural architectures for grasp synthesis.
• “DCoG/”: DexCoGModel class and associated files for the complete inference pipeline.
• “mano/”: MANO hand model code and parameters.
• “dataprocess.py” + “utils/”: dataset loading, point-cloud sampling, loss functions, and visualization.
• “run_complete.py” & “run_only_opt.py”: entry scripts for full pipeline vs. optimization-only pipeline.
• “config.json” & “config.py”: global configuration for paths, hyperparameters, and dependencies.
• “sample/”: example input point clouds and meshes.
• “.imgs/”: sample output images.
Pull extra insights from the README
• Training requires the OBMan and ShapeNetCore.v2 datasets and produces model checkpoints under PointCVAE/model.pth and SupNet/model.pth.
• Inference uses pre-trained models and the MANO RIGHT hand mesh data.
• Dependencies: PyTorch, PyTorch3D, Open3D, Trimesh, NumPy, MANO.
Main components & interactions
A. Data Layer
• External datasets: OBMan (hand-object interactions) and ShapeNetCore.v2 (object meshes)
• Sample inputs: point clouds (.npy), mesh (.obj)
B. Data Processing
• dataprocess.py + utils/load_obman.py: point-cloud sampling, annotation loading
• utils/tools.py, utils/loss.py, utils/vis.py
C. Model Training
• PointCVAE
– network modules under PointCVAE/network (pointnet_encoder, cave, point_cvae_net)
– train.py, obman_dataset.py → produces PointCVAE/model.pth
• SupNet
– network modules under SupNet/network (pointnet_encoder, supnet)
– train.py, obman_dataset.py → produces SupNet/model.pth
D. MANO Hand Model
• mano/model.py, lbs.py, joints_info.py, utils.py → loads/manipulates hand mesh for forward kinematics & skinning
E. Inference & Optimization
• DCoGModel (DCoG/DCoGModel.py) orchestrates:
– loading object point clouds
– initializing grasps from SupNet and/or PointCVAE
– optimizing contact via “only_opt” or “complete” pipelines
• run_complete.py: full pipeline (initialization + optimization)
• run_only_opt.py: optimization only, given an initial pose
F. Visualization & Output
• utils/vis.py, Open3D & Trimesh to render final hand-object meshes/point clouds
• Outputs saved as .png and .npy under sample workflows
Architectural patterns & principles
• Layered architecture: Data → Processing → Model Training/Inference → Visualization
• Modular separation of concerns: two independent generative networks (PointCVAE, SupNet) share dataset and utility code.
• Model-driven flow: trained checkpoints feed into inference module (DCoGModel).
• Config-driven deployment: single config.json toggles dataset paths & hyperparameters.
Essential elements
• Clear, descriptive labels on all components (e.g., “PointCVAE Training Module,” “MANO Hand Model”).