How hmxiong/3ur-llm works
The project, “3UR-LLM,” is an end-to-end multimodal large language model primarily aimed at 3D scene understanding. In other words, it leverages vision–language modeling techniques and integrates 3D-specific components (e.g., ThreeDETR and PointNet2) into a unified framework to perform tasks involving both language and 3D scene data.
Main Components Identification
• Data Ingestion & Processing:
• Input pipelines that load and preprocess 3D scene data as well as conventional imagery and text.
• Modules within the “lavis/datasets” and “lavis/processors” directories handle data builders, dataset configurations, and transforms.
• Model Architecture:
• Multiple model variants exist under “lavis/models” in both the BLIP2-base and BLIP2_LoRA setups.
• The “blip2_models” subfolder includes several components (e.g., Qformer, blip2, variants built on T5, OPT, and specialized 3D modules such as ThreeDETR and its concate, LoRA-specific variants).
• There are also integrated components for point cloud processing (PointNet2) and 3D detection (3D-TR, possibly combined with additional modules from EPCL in the LoRA variant).
• Training & Optimization:
• Training scripts, configuration files, and runner modules (found in “lavis/projects”, “scripts”, and “runners”) are responsible for pretraining, finetuning, and model evaluation.
• Multiple YAML configuration files under “configs/datasets” and “configs/models” outline hyperparameters and dataset specifics.
• Inference & Evaluation:
• “inference.py” and “evaluate.py” provide the endpoints to perform model inference and scoring.
• Scoring utilities (in “calculate_scores”) and test scripts (e.g., “test.json”, “test_vdetr.sh”) indicate the post-processing and evaluation mechanics.
• Utility Components:
• Common utilities such as configuration management, logging, and distributed utilities are located within “lavis/common”.
• Helper scripts (in “scripts” directories) facilitate launching various training experiments and deploying inference services.
Relationships and Interactions
• Data flows from the ingestion layers (using dataset builders and processors) into the model training pipeline.
• The models, whether they are the base BLIP2 modules or LoRA-tuned variants, receive processed inputs and generate multimodal outputs.
• Training runners take care of iterating over data, managing distributed training (e.g., via “runner_iter.py”, “runner_base.py”), and triggering evaluation scripts.
• The inference and scoring modules link back to the trained model, assessing performance and integrating with external API layers if applicable.
• There is an integration layer with external benchmarks or utilities (for example, the FastChat-related modules and LLM judge in “lavis/models/fastchat”).
Architectural Patterns and Design Principles
• Modularity is a key design principle: different components such as data processing, modeling, and utility functions are clearly separated.
• The system uses configuration-driven experimentation: YAML files and config modules control dataset parameters and model hyperparameters.
• There is an abstraction for distributed processing and training (found in “lavis/common/dist_utils.py” and the runners modules).
Technologies, Frameworks, and Libraries
• Python-based deep learning ecosystem (with custom modules likely built atop PyTorch).
• Custom implementations for multimodal integration including aspects such as Qformer and 3D detection (possibly leveraging libraries like PointNet2 for point cloud data).
• Utility libraries for distributed training and logging.
• Scripts for launching training, pretraining, and inference – which may hint at container orchestration or distributed training workflows.
Elements
• A block representing the data sources (raw 3D scenes, images, text), leading into a “Data Processing & Dataset Builders” block.
• One or more blocks for “Preprocessing/Transformers” (derived from lavis/processors).
• A central “Model Engine” block subdivided into:
• The BLIP2 base model and its derivatives.
• Specialized 3D modules (ThreeDETR, PointNet2 integration).
• LoRA tuning variants under the LoRA branch.
• Blocks for “Training & Optimization” (runners, scripts, and config managers) linking to and from the model.
• Separate blocks for “Inference” and “Evaluation/Score Calculation” (linking to calculate_scores utilities and evaluate.py).
• Annotations for configuration files (YAML files) and utility modules which drive the modularity.