# tangyuan96/minigpt-3d architecture

> Architecture diagram of the GitHub repository tangyuan96/minigpt-3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tangyuan96/minigpt-3d
- Repository: https://github.com/tangyuan96/minigpt-3d

## Overview

##### Project type and purpose
- MiniGPT-3D is a research‐oriented, full‐stack machine‐learning system for aligning 3D point clouds with large language models (LLMs).
- It provides modules for data preparation, multi‐stage training, evaluation (OpenAI or open‐source Qwen API), and a Gradio demo UI.

##### File structure
- Top‐level scripts:
- train.py – orchestrates the four‐stage training pipeline (configs in train_configs/)
- UI_demo.py – launches the Gradio conversation interface (using eval_configs/)
- Core package “minigpt4/”: common utilities, config, multi‐modal model definitions, training runners, tasks, data builders, processors.
- “pointllm/”: evaluation scripts and data utilities for object classification/captioning benchmarks.
- Configuration directories:
- train_configs/ – YAMLs for each training stage
- eval_configs/ – YAMLs for benchmarking and demo
- External‐weight placeholders: params_weight/ (LLM weights, point encoder weights, etc.)

##### README for additional insights
- Four‐stage cascaded training that transfers knowledge from 2D LLMs to 3D LLM.
- Data sources: Objaverse point clouds, ModelNet40 test set, annotation JSONs/TXT.
- Evaluation pipelines: open‐vocabulary classification, closed‐set classification, object captioning.
- Demo: Gradio UI that wraps inference scripts with a simple web front end.

- Data Ingestion & Preprocessing: minigpt4/datasets builders + pointllm/data_utils
- Model Core:
– Point‐Cloud Encoder (PointTransformer in minigpt4/models/pointbert)
– Q-Former & projector (minigpt4/models/Qformer.py)
– LLM backbone (Phi, TinyGPT variants via HuggingFace weights in params_weight/)
– Mixture of Query Experts (implements feature aggregation)
- Training Pipeline: train.py + train_configs/ → four Stage modules (Stage I–IV)
- Evaluation Services:
– Benchmark Scripts (pointllm/eval/*.py)
– API‐based evaluators (OpenAI, Qwen2 via evaluator_opensource_llm_QwenAPI.py)
– Traditional metrics evaluator (traditional_evaluator.py)
- Demo UI: UI_demo.py + Gradio (eval_configs/MiniGPT_3D_conv_UI_demo.yaml)
- Configuration Manager: YAML files under configs/ and train_configs/, eval_configs/
- Utilities & Common: minigpt4/common (logging, distributed utils, registry, vqa_tools)

1. Data Layer → Data Ingestion & Preprocessing
2. Preprocessed data → Training Pipeline (Stage I–IV)
3. Training Pipeline → Model Core (updates weights in params_weight/)
4. Model Core + Configs → Evaluation Services (input: test splits, prompts)
5. Model Core + Configs → Demo UI (user uploads point cloud or enters Objaverse ID)
6. Evaluation Services → Output JSONs or metrics logs
7. Demo UI → Gradio Front End → User browser

c. Architectural patterns & design principles
- Pipeline pattern: staged training, clearly separated by YAML config.
- Modular design: separate modules for data, models, training tasks, evaluation, UI.
- Config‐driven: every component is parameterized by a YAML file.
- Reuse & extensibility: common utilities, registry patterns, plugin‐like dataset and model builders.

d. Key technologies & frameworks
- PyTorch 2.0 for model implementation & distributed training (minigpt4/common/dist_utils.py)
- Hugging Face Transformers for LLM weights
- Gradio for interactive demo
- YAML for configuration
- OpenAI / Qwen2 API for evaluation
- CUDA/CuDNN for GPU acceleration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        Objaverse["Objaverse"]:::data
        ModelNet40["ModelNet40"]:::data
        Annotations["Annotations"]:::data
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DatasetBuilders["Minigpt4 Dataset Builders"]:::code
        DataUtils["Minigpt4 Data Utils"]:::code
        PointLLMUtils["PointLLM Data Utils"]:::code
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        TrainScript["train.py"]:::code
        subgraph "Stages"
            Stage1["Stage 1: 2D-to-3D Projection Training"]:::code
            Stage2["Stage 2: Expert Aggregation Fine-tune"]:::code
            Stage3["Stage 3: Multi-view Alignment"]:::code
            Stage4["Stage 4: Final Fine-tune"]:::code
        end
    end

    %% Model Core
    subgraph "Model Core"
        PointEncoder["Point-Cloud Encoder\n(PointTransformer)"]:::code
        QFormer["Q-Former & Projector"]:::code
        LLMConfigs["LLM Backbone\n(Phi, TinyGPT variants)"]:::code
    end

    %% Configuration Manager
    ConfigManager["Configuration Manager\n(YAML files)"]:::code

    %% Utilities & Common
    Utilities["Utilities & Common"]:::code

    %% Evaluation Services
    subgraph "Evaluation Services"
        Benchmark["Benchmark Scripts"]:::service
        APIEvaluator["API-based Evaluators\n(OpenAI, Qwen)"]:::service
        TraditionalEval["Traditional Metrics Evaluator"]:::service
    end
    OutputMetrics["Output JSONs / Metrics Logs"]:::data

    %% Demo UI
    subgraph "Demo UI"
        UILauncher["UI_demo.py"]:::ui
        UIConfig["UI Config"]:::code
        GradioFront["Gradio Front End"]:::ui
        UserBrowser["User Browser"]:::ui
    end

    %% Flows
    Objaverse --> DatasetBuilders
    ModelNet40 --> DatasetBuilders
    Annotations --> DataUtils
    DatasetBuilders --> PointLLMUtils
    DataUtils --> TrainScript
    PointLLMUtils --> TrainScript

    TrainScript --> Stage1
    TrainScript --> Stage2
    TrainScript --> Stage3
    TrainScript --> Stage4

    Stage4 --> PointEncoder
    Stage4 --> QFormer
    Stage4 --> LLMConfigs

    PointEncoder --> APIEvaluator
    QFormer --> Benchmark
    LLMConfigs --> TraditionalEval

    ConfigManager --> TrainScript
    ConfigManager --> APIEvaluator
    ConfigManager --> UILauncher

    Utilities --> TrainScript
    Utilities --> APIEvaluator
    Utilities --> UILauncher

    APIEvaluator --> OutputMetrics
    Benchmark --> OutputMetrics
    TraditionalEval --> OutputMetrics

    LLMConfigs --> UILauncher
    UILauncher --> UIConfig
    UIConfig --> GradioFront
    GradioFront --> UserBrowser

    %% Click Events
    click Objaverse "https://github.com/tangyuan96/minigpt-3d/tree/main/minigpt4/configs/datasets/Objaverse"
    click ModelNet40 "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/data/modelnet_config/ModelNet40.yaml"
    click DatasetBuilders "https://github.com/tangyuan96/minigpt-3d/tree/main/minigpt4/datasets/builders/"
    click DataUtils "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/datasets/data_utils.py"
    click PointLLMUtils "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/data/utils.py"
    click PointLLMUtils "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/data/object_point_dataset.py"
    click PointEncoder "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/models/pointbert/point_encoder.py"
    click QFormer "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/models/Qformer.py"
    click LLMConfigs "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/models/configuration_phi.py"
    click LLMConfigs "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/models/modeling_phi.py"
    click LLMConfigs "https://github.com/tangyuan96/minigpt-3d/blob/main/minigpt4/models/modeling_llama.py"
    click TrainScript "https://github.com/tangyuan96/minigpt-3d/blob/main/train.py"
    click Stage1 "https://github.com/tangyuan96/minigpt-3d/blob/main/train_configs/MiniGPT_3D/stage_1.yaml"
    click Stage2 "https://github.com/tangyuan96/minigpt-3d/blob/main/train_configs/MiniGPT_3D/stage_2.yaml"
    click Stage3 "https://github.com/tangyuan96/minigpt-3d/blob/main/train_configs/MiniGPT_3D/stage_3.yaml"
    click Stage4 "https://github.com/tangyuan96/minigpt-3d/blob/main/train_configs/MiniGPT_3D/stage_4.yaml"
    click Benchmark "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/eval/eval_modelnet_cls.py"
    click Benchmark "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/eval/eval_objaverse.py"
    click APIEvaluator "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/eval/evaluator_opensource_llm_QwenAPI.py"
    click TraditionalEval "https://github.com/tangyuan96/minigpt-3d/blob/main/pointllm/eval/traditional_evaluator.py"
    click UILauncher "https://github.com/tangyuan96/minigpt-3d/blob/main/UI_demo.py"
    click UIConfig "https://github.com/tangyuan96/minigpt-3d/blob/main/eval_configs/MiniGPT_3D_conv_UI_demo.yaml"
    click ConfigManager "https://github.com/tangyuan96/minigpt-3d/tree/main/minigpt4/configs/"
    click Utilities "https://github.com/tangyuan96/minigpt-3d/tree/main/minigpt4/common/"

    %% Styles
    classDef code fill:#9cf,stroke:#333,stroke-width:1px
    classDef data fill:#9f6,stroke:#333,stroke-width:1px
    classDef service fill:#fc9,stroke:#333,stroke-width:1px
    classDef ui fill:#c9f,stroke:#333,stroke-width:1px
```
