# hmxiong/3ur-llm architecture

> Architecture diagram of the GitHub repository hmxiong/3ur-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hmxiong/3ur-llm
- Repository: https://github.com/hmxiong/3ur-llm

## Overview

The project, “3UR-LLM,” is an end-to-end multimodal large language model primarily aimed at 3D scene understanding. In other words, it leverages vision–language modeling techniques and integrates 3D-specific components (e.g., ThreeDETR and PointNet2) into a unified framework to perform tasks involving both language and 3D scene data.

##### Main Components Identification
- Data Ingestion & Processing:
- Input pipelines that load and preprocess 3D scene data as well as conventional imagery and text.
- Modules within the “lavis/datasets” and “lavis/processors” directories handle data builders, dataset configurations, and transforms.
- Model Architecture:
- Multiple model variants exist under “lavis/models” in both the BLIP2-base and BLIP2_LoRA setups.
- The “blip2_models” subfolder includes several components (e.g., Qformer, blip2, variants built on T5, OPT, and specialized 3D modules such as ThreeDETR and its concate, LoRA-specific variants).
- There are also integrated components for point cloud processing (PointNet2) and 3D detection (3D-TR, possibly combined with additional modules from EPCL in the LoRA variant).
- Training & Optimization:
- Training scripts, configuration files, and runner modules (found in “lavis/projects”, “scripts”, and “runners”) are responsible for pretraining, finetuning, and model evaluation.
- Multiple YAML configuration files under “configs/datasets” and “configs/models” outline hyperparameters and dataset specifics.
- Inference & Evaluation:
- “inference.py” and “evaluate.py” provide the endpoints to perform model inference and scoring.
- Scoring utilities (in “calculate_scores”) and test scripts (e.g., “test.json”, “test_vdetr.sh”) indicate the post-processing and evaluation mechanics.
- Utility Components:
- Common utilities such as configuration management, logging, and distributed utilities are located within “lavis/common”.
- Helper scripts (in “scripts” directories) facilitate launching various training experiments and deploying inference services.

##### Relationships and Interactions
- Data flows from the ingestion layers (using dataset builders and processors) into the model training pipeline.
- The models, whether they are the base BLIP2 modules or LoRA-tuned variants, receive processed inputs and generate multimodal outputs.
- Training runners take care of iterating over data, managing distributed training (e.g., via “runner_iter.py”, “runner_base.py”), and triggering evaluation scripts.
- The inference and scoring modules link back to the trained model, assessing performance and integrating with external API layers if applicable.
- There is an integration layer with external benchmarks or utilities (for example, the FastChat-related modules and LLM judge in “lavis/models/fastchat”).

##### Architectural Patterns and Design Principles
- Modularity is a key design principle: different components such as data processing, modeling, and utility functions are clearly separated.
- The system uses configuration-driven experimentation: YAML files and config modules control dataset parameters and model hyperparameters.
- There is an abstraction for distributed processing and training (found in “lavis/common/dist_utils.py” and the runners modules).

##### Technologies, Frameworks, and Libraries
- Python-based deep learning ecosystem (with custom modules likely built atop PyTorch).
- Custom implementations for multimodal integration including aspects such as Qformer and 3D detection (possibly leveraging libraries like PointNet2 for point cloud data).
- Utility libraries for distributed training and logging.
- Scripts for launching training, pretraining, and inference – which may hint at container orchestration or distributed training workflows.

##### Elements
- A block representing the data sources (raw 3D scenes, images, text), leading into a “Data Processing & Dataset Builders” block.
- One or more blocks for “Preprocessing/Transformers” (derived from lavis/processors).
- A central “Model Engine” block subdivided into:
- The BLIP2 base model and its derivatives.
- Specialized 3D modules (ThreeDETR, PointNet2 integration).
- LoRA tuning variants under the LoRA branch.
- Blocks for “Training & Optimization” (runners, scripts, and config managers) linking to and from the model.
- Separate blocks for “Inference” and “Evaluation/Score Calculation” (linking to calculate_scores utilities and evaluate.py).
- Annotations for configuration files (YAML files) and utility modules which drive the modularity.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion & Processing"
        DS["Raw Data: 3D Scenes, Images, Text"]:::data
        DB["Data Processing & Dataset Builders"]:::data
        PT["Preprocessing/Transformers"]:::data
    end

    subgraph "Model Engine"
        M["Core BLIP2 Models"]:::model
        BM["BLIP2 Models (blip2_models)"]:::model
        T["3D-specific: ThreeDETR"]:::model
        P2["Point Cloud Processing: PointNet2"]:::model
    end

    subgraph "Training & Optimization"
        TS["Training Scripts & Project Configurations"]:::train
        TR["Training Runner Modules"]:::train
        Y["YAML Configuration Files"]:::train
    end

    subgraph "Inference & Evaluation"
        I["Inference Endpoint"]:::infer
        E["Evaluation Endpoint"]:::infer
        SC["Scoring Utilities"]:::infer
    end

    subgraph "Utility Components"
        U["Common Utilities"]:::util
    end

    %% Data flow connections
    DS --> DB
    DB --> PT
    PT --> M

    %% Model Engine internal structure
    M --> BM
    M --> T
    M --> P2

    %% Training and Optimization flow
    M --> TS
    TS --> TR
    TS --> Y

    %% Inference and Evaluation flow
    TS --> I
    I --> E
    I --> SC

    %% Utility dependencies (dotted lines)
    U -.-> M
    U -.-> TS

    %% Click Events for Data Ingestion & Processing
    click DB "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/datasets"
    click PT "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/processors"

    %% Click Events for Model Architecture
    click M "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/models"
    click BM "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/models/blip2_models"
    click T "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/models/ThreeDETR"
    click P2 "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/models/pointnet2"

    %% Click Events for Training & Optimization
    click TS "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/projects"
    click TR "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/runners"
    click Y "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/configs"

    %% Click Events for Inference & Evaluation
    click I "https://github.com/hmxiong/3ur-llm/blob/main/3DLLM_BLIP2-base/inference.py"
    click E "https://github.com/hmxiong/3ur-llm/blob/main/3DLLM_BLIP2-base/evaluate.py"
    click SC "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/calculate_scores"

    %% Click Event for Utility Components
    click U "https://github.com/hmxiong/3ur-llm/tree/main/3DLLM_BLIP2-base/lavis/common"

    %% Style Classes
    classDef data fill:#AED581,stroke:#33691E,stroke-width:2px;
    classDef model fill:#64B5F6,stroke:#0D47A1,stroke-width:2px;
    classDef train fill:#FFB74D,stroke:#E65100,stroke-width:2px;
    classDef infer fill:#BA68C8,stroke:#4A148C,stroke-width:2px;
    classDef util fill:#B0BEC5,stroke:#263238,stroke-width:2px;
```
