How djiajunustc/3d-llava works
Project type and purpose
• 3D-LLaVA is a research-grade, full‐stack deep‐learning system for 3D vision–language modeling. It ingests 3D point‐cloud data plus text prompts and supports training, fine-tuning (via LoRA), evaluation (VQA, dense captioning, referring segmentation), and serving (CLI, Gradio).
• Its core is the Omni Superpoint Transformer (OST) integrated into a LLaVA‐style multimodal architecture.
File structure
Top‐level components:
• checkpoints: serialized pretrained model weights
• libs: two CUDA‐accelerated custom ops libraries (pointgroup_ops, pointops)
• llava: Python package with modules for
– model building (builder.py, llava_arch, language and multimodal encoder, OST, projector)
– training (train.py, llava_trainer, monkey patches)
– evaluation (eval scripts per task)
– serving (CLI, controller, Gradio server, worker processes)
• scripts: shell scripts for training and multi-GPU evaluation
README for additional insights
– The “Overview” contains the high‐level framework.png showing data flow: point cloud + text → OST → LLaVA language model → output
– Dependencies: Docker image, Hugging Face datasets/checkpoints
– Workflows: data placement in playground/, LoRA fine‐tuning, multi-GPU evaluation
1. Data Layer
– Scannet / text datasets (placed under playground/data)
– Checkpoints repository (checkpoints/pc_pretrained/*.pth)
2. Preprocessing & Custom Ops
– pointgroup_ops (3D clustering & voxelization CUDA kernels)
– pointops (sampling, grouping, attention CUDA kernels)
3. Core Model
– Omni Superpoint Transformer (OST)
• Visual feature selector
• Visual prompt encoder
• Referring mask decoder
– Multimodal encoder (CLIP + OST + projector)
– LLaMA‐based language model (via llava_llama.py)
– LoRA adapters (apply_delta.py + make_delta.py)
4. Training
– llava/train/train.py, llava_trainer (with flash_attn / xformers patches)
– LoRA fine-tuning script (scripts/train/finetune-3d-llava-lora.sh)
5. Evaluation
– Task‐specific evaluators in llava/eval/
– Multi-GPU shell scripts under scripts/eval/
6. Serving
– Controller & CLI (llava/serve/controller.py, cli.py)
– Worker processes (model_worker.py, sglang_worker.py)
– Gradio web server (gradio_web_server.py)
b. Relationships & interactions
– Data Layer feeds into Preprocessing & Custom Ops
– Preprocessed features + text prompt go into Core Model
– Core Model parameters loaded from checkpoints; training updates via LoRA
– Training component writes new checkpoints
– Evaluation component loads checkpoints, invokes Core Model, outputs metrics
– Serving component loads checkpoints, listens for queries, dispatches to model_worker → returns responses
c. Architectural patterns & principles
– Modular separation: custom ops libraries isolated under libs/
– Plugin‐style encoders & decoders via registry (pc_utils/registry.py)
– LoRA fine‐tuning adapters decoupled from base weights (apply_delta, make_delta)
– Multi-process serving controller + workers (scalable design)
d. Key technologies & frameworks
– Python + PyTorch
– CUDA custom ops (pointgroup_ops, pointops)
– LoRA (low‐rank adapters)
– FlashAttention / xFormers patches
– Docker for environment
– Gradio for web serving