# 80chen86/ipdn architecture

> Architecture diagram of the GitHub repository 80chen86/ipdn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/80chen86/ipdn
- Repository: https://github.com/80chen86/ipdn

## Overview

##### Overall project purpose and type
- This project is a deep learning research codebase for 3D referring expression segmentation. It leverages both 3D point cloud data and multi-view image features (via CLIP) to enhance segmentation performance.
- The system is full-stack in the sense that it covers data preparation, model building (including custom CUDA/C++ operations), training, and inference—even though everything runs under Python/PyTorch.

##### Major architectural layers/components
a. Data preparation and input pipeline:
- Data directories (data/scannetv2, data/ScanRefer, data/Multi3DRefer) and corresponding preprocessing scripts (e.g., prepare_data.sh, image2point_clip.py) suggest an initial data ingestion and processing layer.
- This includes splitting & preprocessing point cloud data, extracting image features using CLIP, and preparing annotations.
- External data dependencies (e.g., the ScanNet dataset, ScanRefer and Multi3DRefer annotations).

b. Core processing libraries:
- The “ipdn” folder hosts the heart of the model including:
- The dataset classes (ipdn/dataset) that define how the point cloud and 2D images are read/decoded.
- The model components (ipdn/model) including backbone, decoding (Prompt-Aware Decoder), and loss functions.
- Custom CUDA/C++ operations are implemented in ipdn/lib/pointgroup_ops. Represent these as specialized acceleration libraries for grouping and pooling operations in the network.
- The “pointnet2” folder contains extensions for point cloud processing (sampling, grouping, interpolation) which are integrated into the model pipeline.
- “sng_parser” is likely used for parsing scene graphs or natural language expressions to build the prompt-based guidance that the model uses.

c. Deep learning and external framework components:
- The project is built on top of PyTorch (as indicated by the “ipdn/torch” subfolder that customizes or wraps some of PyTorch’s functionalities) and makes use of Pytorch3D for potential 3D data handling.
- External deep learning dependencies such as spconv and additional libraries from GitHub repositories (e.g., segmentator) should be noted as auxiliary services or library dependencies.

d. Training and inference pipelines:
- The scripts folder (scripts/train_gres.sh, scripts/train_res.sh, scripts/test_gres.sh, scripts/test_res.sh) drives both training and inference.
- “tools” which hosts helper scripts like test.py and train.py that might be used for debugging or additional experiment management.

##### Interactions and data flow between these components
- How raw data (point clouds, images, annotations) is ingested from external sources and prepared by the “data” processing modules.
- Next, indicate that the processed data are fed into dataset classes provided in the ipdn/dataset module.
- The training pipeline (initiated via scripts or tools) loads data and then passes it to the core model (ipdn/model) which ties together the backbone (possibly using pointnet2 for low-level feature extraction) and the innovative modules (MSE and PAD).
- Within the model, custom modules from ipdn/lib/pointgroup_ops are used for efficient computation, with some parts compiled from CUDA/C++.
- The output (segmentation predictions) is then evaluated against ground truth from the annotations, and the results are used for both training loss computation and eventual inference.
- External dependencies (like segmentator and Pytorch3D) as supporting components that are integrated into the core pipeline.

##### Architectural patterns and important design principles
- The design is modular. Each folder corresponds to a distinct function: data handling, model definition, optimization/training, library extensions, and experiment scripts.
- Separation of concerns between data preprocessing, core model logic, customized operations (integrated via a C++/CUDA extension mechanism), and the training interfaces.
- The usage of multi-view image features and language-based prompt decoding implies a multi-input architecture where data streams (3D features, 2D image tokens, and language features) converge in the network.

- Below that, show the Data Processing Layer (scripts in data/ and parts of sng_parser if used for linguistic parsing).
- On one side, the Dataset Loader module from ipdn/dataset.
- A sub-part for “Prompt-Aware Decoder” and “Multi-view Semantic Embedding” as key innovations.
- At the bottom, emphasize the Training and Inference Engine, driven by scripts in the scripts/ folder and tools in tools/.

##### Detail
- Avoid over-summarization: while high-level flow is critical, capturing these individual modules/components will help when discussing integration and debugging with the complete team.

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Data Sources
    subgraph "External Data Sources"
        ES1["ScanNet Dataset"]:::external
        ES2["ScanRefer Annotations"]:::external
        ES3["Multi3DRefer Annotations"]:::external
        ES4["CLIP Extracted Features"]:::external
    end

    %% Data Preprocessing Layer
    subgraph "Data Preprocessing"
        DP1["ScanNet Preprocessing"]:::python
        DP2["Image-to-Point Conversion"]:::python
    end

    %% Dataset Loader
    DL["Dataset Loader"]:::python

    %% Core Model Components
    subgraph "Core Model"
        MD["Model Definitions"]:::python
        CUDA["Custom CUDA Ops"]:::cuda
        PCO["Point Cloud Ops"]:::python
        LP["Language Parsing"]:::python
        TU["Torch Utilities"]:::python
    end

    %% Training & Inference Pipelines
    subgraph "Training & Inference"
        TI1["Training Scripts"]:::script
        TI2["Experiment Tools"]:::script
    end

    %% External Dependencies
    ED["External DL Libraries"]:::externalLib

    %% Connections from External Data Sources to Data Preprocessing
    ES1 -->|"provides data"| DP1
    ES2 -->|"provides annotations"| DP1
    ES3 -->|"provides annotations"| DP1
    ES4 -->|"provides features"| DP2

    %% Connections from Data Preprocessing to Dataset Loader
    DP1 -->|"processed data"| DL
    DP2 -->|"processed data"| DL

    %% Dataset Loader feeds into Core Model
    DL -->|"loads data"| MD

    %% Core Model integrates specialized operations
    MD -->|"integrates"| CUDA
    MD -->|"integrates"| PCO
    MD -->|"integrates"| LP
    MD -->|"integrates"| TU

    %% External Dependencies support the Core Model
    ED -->|"supports"| MD

    %% Core Model output is used for Training & Inference
    MD -->|"trains & infers"| TI1
    MD -->|"trains & infers"| TI2

    %% Styles and Classes
    classDef external fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef python fill:#ABEBC6,stroke:#196F3D,stroke-width:2px;
    classDef cuda fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef externalLib fill:#F5CBA7,stroke:#943126,stroke-width:2px;
    classDef script fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;

    %% Click Events
    click DP1 "https://github.com/80chen86/ipdn/blob/main/data/scannetv2/prepare_data.sh"
    click DP2 "https://github.com/80chen86/ipdn/blob/main/data/image2point_clip.py"
    click DL "https://github.com/80chen86/ipdn/tree/main/ipdn/dataset/"
    click MD "https://github.com/80chen86/ipdn/blob/main/ipdn/model/ipdn.py"
    click CUDA "https://github.com/80chen86/ipdn/tree/main/ipdn/lib/pointgroup_ops/"
    click PCO "https://github.com/80chen86/ipdn/tree/main/pointnet2/"
    click LP "https://github.com/80chen86/ipdn/blob/main/sng_parser/parser.py"
    click TU "https://github.com/80chen86/ipdn/tree/main/ipdn/torch/"
    click TI1 "https://github.com/80chen86/ipdn/blob/main/scripts/train_gres.sh"
    click TI2 "https://github.com/80chen86/ipdn/blob/main/tools/train.py"
```
