How qizekun/shapellm works
Principal components
• Core Model Modules:
• ReCon++ / Point Encoder: Show the component implemented in ReConV2/models and its integration with pre-training configurations. Also note the integration of extension libraries (the chamfer_distance extension in C++/CUDA).
• Training and Pretraining Pipelines: Represent the training scripts (under scripts, ReConV2/scripts, and llava/train) and how they invoke different training paradigms (e.g., generative pretraining, contrastive learning, finetuning stages, and visual instruction tuning).
• Evaluation and Inference: Show evaluation pipelines (scripts/eval files, playground data for evaluation, and the CLI/gradio serving under llava/serve) which run tasks such as zero-shot learning, few-shot learning, and specific benchmarks (3D MM-Vet, GApartNet).
• External Tools & Utilities: Represent other utilities like point visualization (from the external PointVisualization repo), common utility modules (ReConV2/utils), and configuration parsing.
Relationships and interactions
• Component Interactions: Show that the training pipeline makes use of scripts and tools (runner_pretrain, runner_finetune, runner_zeroshot) that orchestrate the model training and evaluation. Indicate interaction between core model components (ReCon++ based encoder & transformer) and the language model parts (llava/language_model).
• Deployment: Connect the trained model to the serving components (CLI interface in llava/serve, gradio based web server, and other inference scripts) – the inference system consumes point cloud inputs and outputs interactive multi-modal results.
Architectural patterns and design principles
• Modular Design: Emphasize the separation of concerns by isolating configurations, datasets, model definitions, training utilities, and evaluation scripts.
• Pipeline and Stage Separation: Clearly show different stages of training – initial feature alignment (pretraining), followed by full fine-tuning. Also separate evaluation and inference stages.
• Extensibility: Note how additional external evaluation datasets (3D-MM-Vet, GApartNet) and third-party modules like PointNet++ or PointVisualizaiton integrate into the pipeline.
• Integration of Low-Level Optimizations: Include how critical performance modules (like chamfer_distance implemented in C++/CUDA) plug into the overall model design.
Relevant technologies and frameworks
• Python with deep learning frameworks (e.g., PyTorch implied by usage of flash-attn, torch libraries in extensions)
• YAML for configuration management, and scripting (shell scripts) for orchestrating training and evaluation.
• External libraries: Pointnet2_ops from GitHub, flash-attn, and other specialized tools.