# djiajunustc/3d-llava architecture

> Architecture diagram of the GitHub repository djiajunustc/3d-llava, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/djiajunustc/3d-llava
- Repository: https://github.com/djiajunustc/3d-llava

## Overview

##### Project type and purpose
- 3D-LLaVA is a research-grade, full‐stack deep‐learning system for 3D vision–language modeling. It ingests 3D point‐cloud data plus text prompts and supports training, fine-tuning (via LoRA), evaluation (VQA, dense captioning, referring segmentation), and serving (CLI, Gradio).
- Its core is the Omni Superpoint Transformer (OST) integrated into a LLaVA‐style multimodal architecture.

##### File structure
Top‐level components:
- checkpoints: serialized pretrained model weights
- libs: two CUDA‐accelerated custom ops libraries (pointgroup_ops, pointops)
- llava: Python package with modules for
– model building (builder.py, llava_arch, language and multimodal encoder, OST, projector)
– training (train.py, llava_trainer, monkey patches)
– evaluation (eval scripts per task)
– serving (CLI, controller, Gradio server, worker processes)
- scripts: shell scripts for training and multi-GPU evaluation

##### README for additional insights
– The “Overview” contains the high‐level framework.png showing data flow: point cloud + text → OST → LLaVA language model → output
– Dependencies: Docker image, Hugging Face datasets/checkpoints
– Workflows: data placement in playground/, LoRA fine‐tuning, multi-GPU evaluation

1. Data Layer
– Scannet / text datasets (placed under playground/data)
– Checkpoints repository (checkpoints/pc_pretrained/*.pth)
2. Preprocessing & Custom Ops
– pointgroup_ops (3D clustering & voxelization CUDA kernels)
– pointops (sampling, grouping, attention CUDA kernels)
3. Core Model
– Omni Superpoint Transformer (OST)
- Visual feature selector
- Visual prompt encoder
- Referring mask decoder
– Multimodal encoder (CLIP + OST + projector)
– LLaMA‐based language model (via llava_llama.py)
– LoRA adapters (apply_delta.py + make_delta.py)
4. Training
– llava/train/train.py, llava_trainer (with flash_attn / xformers patches)
– LoRA fine-tuning script (scripts/train/finetune-3d-llava-lora.sh)
5. Evaluation
– Task‐specific evaluators in llava/eval/
– Multi-GPU shell scripts under scripts/eval/
6. Serving
– Controller & CLI (llava/serve/controller.py, cli.py)
– Worker processes (model_worker.py, sglang_worker.py)
– Gradio web server (gradio_web_server.py)
b. Relationships & interactions
– Data Layer feeds into Preprocessing & Custom Ops
– Preprocessed features + text prompt go into Core Model
– Core Model parameters loaded from checkpoints; training updates via LoRA
– Training component writes new checkpoints
– Evaluation component loads checkpoints, invokes Core Model, outputs metrics
– Serving component loads checkpoints, listens for queries, dispatches to model_worker → returns responses
c. Architectural patterns & principles
– Modular separation: custom ops libraries isolated under libs/
– Plugin‐style encoders & decoders via registry (pc_utils/registry.py)
– LoRA fine‐tuning adapters decoupled from base weights (apply_delta, make_delta)
– Multi-process serving controller + workers (scalable design)
d. Key technologies & frameworks
– Python + PyTorch
– CUDA custom ops (pointgroup_ops, pointops)
– LoRA (low‐rank adapters)
– FlashAttention / xFormers patches
– Docker for environment
– Gradio for web serving

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DL_Data["Scannet / Text Datasets"]:::data
        DL_Checkpoints["Checkpoints Repository"]:::data
    end

    %% Custom Ops
    subgraph "Preprocessing & Custom Ops"
        direction TB
        OPS_PointGroup["pointgroup_ops CUDA Kernels"]:::ops
        OPS_PointOps["pointops CUDA Kernels"]:::ops
    end

    %% Core Model
    subgraph "Core Model"
        direction TB
        subgraph "Omni Superpoint Transformer (OST)"
            direction TB
            OST_Module["OST Module"]:::core
            OST_Builder["OST Builder"]:::core
            OST_InstCrit["Instance Segmentation Criterion"]:::core
            OST_ReferCrit["Referring Segmentation Criterion"]:::core
            OST_Decoder["SA-Only Decoder"]:::core
        end
        Encoder_MM["Multimodal Encoder Entrypoint"]:::core
        Encoder_CLIP["CLIP Visual Encoder"]:::core
        Encoder_PC["Point-Cloud Encoder"]:::core
        Encoder_Prompt["Prompt Encoder"]:::core
        UNet_SPConv["SPConv U-Net Backbone"]:::core
        Projector["Projector"]:::core
        LM_LLaMA["LLaMA-based Language Model"]:::core
        LoRA_Apply["LoRA Adapter (apply_delta)"]:::core
        LoRA_Make["LoRA Adapter (make_delta)"]:::core
        Registry["Plugin Registry"]:::core
    end

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        direction TB
        Train_Main["Main Training Script"]:::train
        Trainer_Patches["Trainer & Monkey Patches"]:::train
        Train_LoRA_Script["LoRA Fine-tuning Script"]:::train
        Eval_Tasks["Task-specific Evaluators"]:::eval
        Eval_Scripts["Multi-GPU Eval Shell Scripts"]:::eval
    end

    %% Serving
    subgraph "Serving"
        direction TB
        Serve_CLI["CLI Interface"]:::serve
        Serve_Controller["Controller (Orchestration)"]:::serve
        Serve_Worker["Model Worker Process"]:::serve
        Serve_SGLang["SGLang Worker Process"]:::serve
        Serve_Gradio["Gradio Web Server"]:::serve
    end

    %% Data Flow
    DL_Data -->|point clouds + text| OPS_PointGroup
    DL_Data -->|point clouds + text| OPS_PointOps
    DL_Checkpoints -->|load weights| OST_Module

    %% Ops to Core
    OPS_PointGroup -->|preprocessed features| OST_Module
    OPS_PointOps -->|preprocessed features| OST_Module

    %% Core internals
    OST_Module --> OST_Builder
    OST_Module --> OST_InstCrit
    OST_Module --> OST_ReferCrit
    OST_Module --> OST_Decoder
    OST_Module --> Encoder_MM
    Encoder_MM --> Encoder_CLIP
    Encoder_MM --> Encoder_PC
    Encoder_MM --> Encoder_Prompt
    Encoder_MM --> UNet_SPConv
    Encoder_MM --> Projector
    Encoder_MM --> LM_LLaMA
    LM_LLaMA --> LoRA_Apply
    LM_LLaMA --> LoRA_Make
    OST_Module --> Registry

    %% Training / Evaluation relationships
    OST_Module -->|features + tokens| Train_Main
    DL_Checkpoints -->|load base| Train_Main
    Train_Main -->|updates| LoRA_Apply
    Train_Main -->|new checkpoints| DL_Checkpoints
    Eval_Tasks -->|invoke| OST_Module
    Eval_Scripts -->|multi-GPU run| Eval_Tasks

    %% Serving relationships
    Serve_Controller --> Serve_Worker
    Serve_Controller --> Serve_SGLang
    Serve_Worker --> OST_Module
    Serve_SGLang --> OST_Module
    Serve_Gradio --> Serve_Controller

    %% Click Events
    click DL_Checkpoints "https://github.com/djiajunustc/3d-llava/blob/main/checkpoints/pc_pretrained/ost-sa-only-llava-align-scannet200.pth"
    click OPS_PointGroup "https://github.com/djiajunustc/3d-llava/tree/main/libs/pointgroup_ops/"
    click OPS_PointOps "https://github.com/djiajunustc/3d-llava/tree/main/libs/pointops/"
    click OST_Module "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/ost_module/omni_superpoint_transformer.py"
    click OST_Builder "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/ost_module/builder.py"
    click OST_InstCrit "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/ost_module/inst_seg_criterion.py"
    click OST_ReferCrit "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/ost_module/refer_seg_criterion.py"
    click OST_Decoder "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/ost_module/sa_only_decoder.py"
    click Encoder_MM "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/builder.py"
    click Encoder_CLIP "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/clip_encoder.py"
    click Encoder_PC "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/pointcloud_encoder.py"
    click Encoder_Prompt "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/prompt_encoder.py"
    click UNet_SPConv "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_encoder/spconv_unet.py"
    click Projector "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/multimodal_projector/builder.py"
    click LM_LLaMA "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/language_model/llava_llama.py"
    click LoRA_Apply "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/apply_delta.py"
    click LoRA_Make "https://github.com/djiajunustc/3d-llava/blob/main/llava/model/make_delta.py"
    click Train_Main "https://github.com/djiajunustc/3d-llava/blob/main/llava/train/train.py"
    click Trainer_Patches "https://github.com/djiajunustc/3d-llava/blob/main/llava/train/llava_trainer.py"
    click Trainer_Patches "https://github.com/djiajunustc/3d-llava/blob/main/llava/train/llama_flash_attn_monkey_patch.py"
    click Trainer_Patches "https://github.com/djiajunustc/3d-llava/blob/main/llava/train/llama_xformers_attn_monkey_patch.py"
    click Train_LoRA_Script "https://github.com/djiajunustc/3d-llava/blob/main/scripts/train/finetune-3d-llava-lora.sh"
    click Eval_Tasks "https://github.com/djiajunustc/3d-llava/tree/main/llava/eval/"
    click Eval_Scripts "https://github.com/djiajunustc/3d-llava/blob/main/scripts/eval/multigpu_eval_*.sh"
    click Serve_CLI "https://github.com/djiajunustc/3d-llava/blob/main/llava/serve/cli.py"
    click Serve_Controller "https://github.com/djiajunustc/3d-llava/blob/main/llava/serve/controller.py"
    click Serve_Worker "https://github.com/djiajunustc/3d-llava/blob/main/llava/serve/model_worker.py"
    click Serve_SGLang "https://github.com/djiajunustc/3d-llava/blob/main/llava/serve/sglang_worker.py"
    click Serve_Gradio "https://github.com/djiajunustc/3d-llava/blob/main/llava/serve/gradio_web_server.py"
    click Registry "https://github.com/djiajunustc/3d-llava/blob/main/llava/pc_utils/registry.py"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085;
    classDef ops fill:#ffe5b4,stroke:#994d00,color:#994d00;
    classDef core fill:#d4edda,stroke:#155724,color:#155724;
    classDef train fill:#f5c6cb,stroke:#721c24,color:#721c24;
    classDef eval fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef serve fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
```
