# qizekun/shapellm architecture

> Architecture diagram of the GitHub repository qizekun/shapellm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qizekun/shapellm
- Repository: https://github.com/qizekun/shapellm

## Overview

##### Principal components
- Core Model Modules:
- ReCon++ / Point Encoder: Show the component implemented in ReConV2/models and its integration with pre-training configurations. Also note the integration of extension libraries (the chamfer_distance extension in C++/CUDA).
- Training and Pretraining Pipelines: Represent the training scripts (under scripts, ReConV2/scripts, and llava/train) and how they invoke different training paradigms (e.g., generative pretraining, contrastive learning, finetuning stages, and visual instruction tuning).
- Evaluation and Inference: Show evaluation pipelines (scripts/eval files, playground data for evaluation, and the CLI/gradio serving under llava/serve) which run tasks such as zero-shot learning, few-shot learning, and specific benchmarks (3D MM-Vet, GApartNet).
- External Tools & Utilities: Represent other utilities like point visualization (from the external PointVisualization repo), common utility modules (ReConV2/utils), and configuration parsing.

##### Relationships and interactions
- Component Interactions: Show that the training pipeline makes use of scripts and tools (runner_pretrain, runner_finetune, runner_zeroshot) that orchestrate the model training and evaluation. Indicate interaction between core model components (ReCon++ based encoder & transformer) and the language model parts (llava/language_model).
- Deployment: Connect the trained model to the serving components (CLI interface in llava/serve, gradio based web server, and other inference scripts) – the inference system consumes point cloud inputs and outputs interactive multi-modal results.

##### Architectural patterns and design principles
- Modular Design: Emphasize the separation of concerns by isolating configurations, datasets, model definitions, training utilities, and evaluation scripts.
- Pipeline and Stage Separation: Clearly show different stages of training – initial feature alignment (pretraining), followed by full fine-tuning. Also separate evaluation and inference stages.
- Extensibility: Note how additional external evaluation datasets (3D-MM-Vet, GApartNet) and third-party modules like PointNet++ or PointVisualizaiton integrate into the pipeline.
- Integration of Low-Level Optimizations: Include how critical performance modules (like chamfer_distance implemented in C++/CUDA) plug into the overall model design.

##### Relevant technologies and frameworks
- Python with deep learning frameworks (e.g., PyTorch implied by usage of flash-attn, torch libraries in extensions)
- YAML for configuration management, and scripting (shell scripts) for orchestrating training and evaluation.
- External libraries: Pointnet2_ops from GitHub, flash-attn, and other specialized tools.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Data & Config Management"]:::config
    B["Dataset & Preprocessing"]:::dataset

    subgraph "Model Architecture"
        C["Point Cloud Encoder (ReCon++ Module)"]:::model
        D["Chamfer Distance Extension"]:::extension
        E["Transformer Module"]:::model
        subgraph "Language & Multimodal Module"
            F["Language Model Components"]:::language
            G["Multimodal Encoder & Projector"]:::language
            H["Overall Llava Architecture"]:::language
        end
    end

    subgraph "Training Pipeline"
        I["Pretraining/Finetuning Scripts (ReConV2/scripts & llava/train)"]:::training
        J["Runner Utilities"]:::training
    end

    subgraph "Evaluation & Inference"
        K["Evaluation Scripts"]:::inference
        L["Serving Interface"]:::inference
    end

    M["External Tools & Utilities"]:::external

    %% Data Flow
    A --> B
    %% Model Architecture from preprocessed data
    B --> C
    B --> E
    %% Internal model relationships
    C --> D
    C --> E
    E --> F
    E --> G
    E --> H
    %% Model integrated into training
    E --> I
    D --> I
    A --> I
    %% Training Pipeline internal flow
    I --> J
    I --> K
    %% Evaluation chain
    K --> L
    %% External utilities support evaluation
    M --> K

    %% Click Events
    click A "https://github.com/qizekun/shapellm/tree/main/ReConV2/cfgs"
    click B "https://github.com/qizekun/shapellm/tree/main/ReConV2/datasets"
    click C "https://github.com/qizekun/shapellm/blob/main/ReConV2/models/ReCon.py"
    click D "https://github.com/qizekun/shapellm/tree/main/ReConV2/extensions/chamfer_distance"
    click E "https://github.com/qizekun/shapellm/blob/main/ReConV2/models/transformer.py"
    click F "https://github.com/qizekun/shapellm/tree/main/llava/model/language_model"
    click G "https://github.com/qizekun/shapellm/tree/main/llava/model/multimodal_encoder"
    click H "https://github.com/qizekun/shapellm/blob/main/llava/model/llava_arch.py"
    click I "https://github.com/qizekun/shapellm/tree/main/ReConV2/scripts"
    click J "https://github.com/qizekun/shapellm/tree/main/ReConV2/tools"
    click K "https://github.com/qizekun/shapellm/tree/main/scripts/eval"
    click L "https://github.com/qizekun/shapellm/tree/main/llava/serve"
    click M "https://github.com/qizekun/shapellm/tree/main/ReConV2/utils"

    %% Styles
    classDef config fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef dataset fill:#90be6d,stroke:#000,stroke-width:2px;
    classDef model fill:#f9844a,stroke:#000,stroke-width:2px;
    classDef extension fill:#f9844a,stroke:#000,stroke-dasharray: 5, 5;
    classDef language fill:#f94144,stroke:#000,stroke-width:2px;
    classDef training fill:#577590,stroke:#000,stroke-width:2px;
    classDef inference fill:#277da1,stroke:#000,stroke-width:2px;
    classDef external fill:#9d4edd,stroke:#000,stroke-width:2px;
```
