How tangyuan96/minigpt-3d works
Project type and purpose
• MiniGPT-3D is a research‐oriented, full‐stack machine‐learning system for aligning 3D point clouds with large language models (LLMs).
• It provides modules for data preparation, multi‐stage training, evaluation (OpenAI or open‐source Qwen API), and a Gradio demo UI.
File structure
• Top‐level scripts:
• train.py – orchestrates the four‐stage training pipeline (configs in train_configs/)
• UI_demo.py – launches the Gradio conversation interface (using eval_configs/)
• Core package “minigpt4/”: common utilities, config, multi‐modal model definitions, training runners, tasks, data builders, processors.
• “pointllm/”: evaluation scripts and data utilities for object classification/captioning benchmarks.
• Configuration directories:
• train_configs/ – YAMLs for each training stage
• eval_configs/ – YAMLs for benchmarking and demo
• External‐weight placeholders: params_weight/ (LLM weights, point encoder weights, etc.)
README for additional insights
• Four‐stage cascaded training that transfers knowledge from 2D LLMs to 3D LLM.
• Data sources: Objaverse point clouds, ModelNet40 test set, annotation JSONs/TXT.
• Evaluation pipelines: open‐vocabulary classification, closed‐set classification, object captioning.
• Demo: Gradio UI that wraps inference scripts with a simple web front end.
• Data Ingestion & Preprocessing: minigpt4/datasets builders + pointllm/data_utils
• Model Core:
– Point‐Cloud Encoder (PointTransformer in minigpt4/models/pointbert)
– Q-Former & projector (minigpt4/models/Qformer.py)
– LLM backbone (Phi, TinyGPT variants via HuggingFace weights in params_weight/)
– Mixture of Query Experts (implements feature aggregation)
• Training Pipeline: train.py + train_configs/ → four Stage modules (Stage I–IV)
• Evaluation Services:
– Benchmark Scripts (pointllm/eval/*.py)
– API‐based evaluators (OpenAI, Qwen2 via evaluator_opensource_llm_QwenAPI.py)
– Traditional metrics evaluator (traditional_evaluator.py)
• Demo UI: UI_demo.py + Gradio (eval_configs/MiniGPT_3D_conv_UI_demo.yaml)
• Configuration Manager: YAML files under configs/ and train_configs/, eval_configs/
• Utilities & Common: minigpt4/common (logging, distributed utils, registry, vqa_tools)
1. Data Layer → Data Ingestion & Preprocessing
2. Preprocessed data → Training Pipeline (Stage I–IV)
3. Training Pipeline → Model Core (updates weights in params_weight/)
4. Model Core + Configs → Evaluation Services (input: test splits, prompts)
5. Model Core + Configs → Demo UI (user uploads point cloud or enters Objaverse ID)
6. Evaluation Services → Output JSONs or metrics logs
7. Demo UI → Gradio Front End → User browser
c. Architectural patterns & design principles
• Pipeline pattern: staged training, clearly separated by YAML config.
• Modular design: separate modules for data, models, training tasks, evaluation, UI.
• Config‐driven: every component is parameterized by a YAML file.
• Reuse & extensibility: common utilities, registry patterns, plugin‐like dataset and model builders.
d. Key technologies & frameworks
• PyTorch 2.0 for model implementation & distributed training (minigpt4/common/dist_utils.py)
• Hugging Face Transformers for LLM weights
• Gradio for interactive demo
• YAML for configuration
• OpenAI / Qwen2 API for evaluation
• CUDA/CuDNN for GPU acceleration