# vobecant/pop3d architecture

> Architecture diagram of the GitHub repository vobecant/pop3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vobecant/pop3d
- Repository: https://github.com/vobecant/pop3d

## Overview

Based on our analysis of both the project’s file tree and the README, this project is a research-oriented computer vision system that integrates 3D occupancy prediction (POP-3D) with open‐vocabulary segmentation capabilities via MaskCLIP. At a high level, the system is built as a modular deep learning framework that leverages data preparation, feature extraction, training, and evaluation pipelines. It integrates contributions from established segmentation frameworks (e.g., MMSegmentation) with custom modules (e.g., TPVFormer and MaskCLIP) to enable open-vocabulary learning from images.

##### Main Components
- Data Management & Preparation:
– Display the nuScenes dataset as the primary data source.
– Include the “data” directories and the scripts (e.g., generate_projections_nuscenes.py, download scripts) that handle dataset downloading, info files management, and projection generation.
- Feature Extraction (MaskCLIP):
– Represent the MaskCLIP module with its own environment and feature extraction pipelines (located in the MaskCLIP folder). Emphasize the conversion of CLIP weights, extraction of image features (via tools/extract_features.py), and the use of pre-trained checkpoints.
- Model Building & Training:
– Highlight the use of common base configurations (found under config/_base_ and MaskCLIP/configs/_base_*) that define models, datasets, and training schedules.
- Evaluation & Retrieval:
– Show the evaluation pipeline (e.g., eval.py, eval_maskclip.py) and the retrieval tool (retrieval.py) for open-vocabulary language-driven retrieval tasks.
- Auxiliary & Utility Modules:
– Represent additional support utilities such as data loaders (in the “dataloader” directory), loss definitions (in both builder/ and mmseg/models/losses), visualization utilities, and scripts that orchestrate distributed training (scripts folder with SLURM-specific scripts).

##### Relationships & Data Flow
- Data Flow:
– Then, indicate how these projections feed into the MaskCLIP feature extraction pipeline.
– Next, show how the extracted features (from MaskCLIP) serve as targets in the training loop for POP-3D.
- Control & Configuration Flow:
– Highlight a configuration management block that “drives” both training (via config files) and model building (mmseg, builder directories).
- Integrated Environments:
– Represent the two distinct conda environments (pop3d and maskclip) and their roles in isolating dependencies.
- Evaluation Branch:

##### Architectural Patterns and Design Principles
- Modularity & Separation of Concerns:
- Reusability:
– The large number of configuration files (in both “config/_base_” and “MaskCLIP/configs/_base_”) indicates a design that emphasizes reusability and flexibility in experimentation.
- Pipeline-Oriented Design:
– The sequential data processing—from data preparation to feature extraction (via MaskCLIP) and then training—should be illustrated as a pipeline.
- Integration of Pre-trained Models:
– Emphasize the integration points for pre-trained checkpoints and external models (e.g., CLIP model weights, MaskCLIP+ weights) in the architecture.

##### Technologies and Frameworks
- Deep Learning Libraries: PyTorch, MMCV, and MMsegmentation (as indicated by the “mmseg” module).
- Data Handling: Custom dataloader modules and dataset wrappers.
- Distributed Training: SLURM launcher scripts and distributed training scripts.
- Model Architectures: Transformer-based models (TPVFormer), classic CNN backbones via maskclip (ResNet, ViT) and segmentation heads (various decode heads).
- Pre-trained Model Integration: Conversion tools found in tools/maskclip_utils and model converters (e.g., mit2mmseg.py) to integrate non-native architectures into mmseg.

##### Specific Elements
- A block labeled “Data Preparation” with subcomponents: nuScenes Dataset, Projection Generation (generate_projections_nuscenes.py), and Info Files.
- A block for “MaskCLIP Module” showing:
- Weight conversion and extraction utilities (tools/maskclip_utils and extract_features.py)
- Configuration files in MaskCLIP/configs (including corrupt variants and web_image examples).
- A block for “Model Building & Training”, with pointers to:
- Configuration management (config/_base_ and MaskCLIP/configs/_base_ folders)
- The training scripts (train.py, scripts/train_slurm.sh, etc.)
- Custom builders and TPVFormer modules.
- A block for “Evaluation & Retrieval” indicating evaluation scripts (eval.py, retrieval.py) and their SLURM alternatives.
- A block for “Utilities and Integration” showing additional modules from “builder”, “dataloader”, “utils”, and “visualization”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation
    subgraph "Data Preparation"
        DS["nuScenes Dataset"]:::dataset
        PG["Projection Generator (generate_projections_nuscenes.py)"]:::script
        DInfo["Download Info Files (scripts/download_info_files.sh)"]:::script
        DRet["Download Retrieval Benchmark (scripts/download_retrieval_benchmark.sh)"]:::script
        DConfig["Dataset Config (config/_base_/dataset_pop3d.py)"]:::config
        LM["Label Mapping (config/label_mapping/nuscenes.yaml)"]:::config
    end

    %% Feature Extraction (MaskCLIP)
    subgraph "Feature Extraction (MaskCLIP)"
        MC["MaskCLIP Module (MaskCLIP/)"]:::maskclip
        EF["Feature Extraction (tools/extract_features.py)"]:::script
        CW1["Convert CLIP Weights (tools/maskclip_utils/convert_clip_weights.py)"]:::script
        CW2["Prompt Engineering (tools/maskclip_utils/prompt_engineering.py)"]:::script
    end

    %% Model Building & Training
    subgraph "Model Building & Training"
        TR["POP-3D Trainer (train.py)"]:::trainer
        CP["Configuration Management (optimizer & schedule)"]:::config
        PC["POP-3D Config (config/pop3d_maskclip_12ep.py)"]:::config
        TP["TPVFormer Module (tpvformer04/)"]:::builder
        CB1["TPV Occupancy Builder (builder/tpv_occupancy_builder.py)"]:::builder
        CB2["TPV LidarSeg Builder (builder/tpv_lidarseg_builder.py)"]:::builder
    end

    %% Evaluation & Retrieval
    subgraph "Evaluation & Retrieval"
        EV["Evaluation (eval.py, eval_maskclip.py)"]:::evaluation
        RET["Retrieval (retrieval.py)"]:::evaluation
        SLURM_EV["SLURM Evaluation (scripts/eval_zeroshot_slurm.sh)"]:::script
    end

    %% Utilities & Integration
    subgraph "Utilities & Integration"
        DL["Data Loaders (dataloader/)"]:::utility
        VIS["Visualization Utilities (visualization/)"]:::utility
        UT["Additional Utilities (utils/)"]:::utility
        DTS1["Train SLURM (scripts/train_slurm.sh)"]:::script
        DTS2["Train PBS (scripts/train_pbs.sh)"]:::script
        DTS3["Launcher (scripts/launcher.sh)"]:::script
        Loss["Loss & Data Builders (builder/loss_builder.py, builder/data_builder.py)"]:::script
    end

    %% Environments
    subgraph "Environments"
        ENV1["pop3d Environment"]:::env
        ENV2["maskclip Environment"]:::env
    end

    %% Data Flow Connections
    DS --> PG
    PG --> DInfo
    PG --> DRet
    DInfo --> DConfig
    DRet --> DConfig
    DConfig --> LM
    LM -->|"toFeatureExtraction"| MC

    MC --> EF
    MC --> CW1
    MC --> CW2

    EF -->|"toTraining"| TR

    CP --> TR
    PC --> TR
    TR --> CB1
    TR --> CB2
    CP --> TP

    TR --> EV
    EV --> RET
    EV --> SLURM_EV

    TR --> UT

    DL --> VIS
    VIS --> UT
    UT --> DTS1
    DTS1 --> DTS2
    DTS2 --> DTS3
    UT --> Loss

    ENV1 --> TR
    ENV2 --> MC

    %% Click Events for Data Preparation
    click PG "https://github.com/vobecant/pop3d/blob/main/generate_projections_nuscenes.py"
    click DInfo "https://github.com/vobecant/pop3d/blob/main/scripts/download_info_files.sh"
    click DRet "https://github.com/vobecant/pop3d/blob/main/scripts/download_retrieval_benchmark.sh"
    click DConfig "https://github.com/vobecant/pop3d/blob/main/config/_base_/dataset_pop3d.py"
    click LM "https://github.com/vobecant/pop3d/blob/main/config/label_mapping/nuscenes.yaml"

    %% Click Events for Feature Extraction
    click MC "https://github.com/vobecant/pop3d/tree/main/MaskCLIP/"
    click EF "https://github.com/vobecant/pop3d/blob/main/tools/extract_features.py"
    click CW1 "https://github.com/vobecant/pop3d/blob/main/tools/maskclip_utils/convert_clip_weights.py"
    click CW2 "https://github.com/vobecant/pop3d/blob/main/tools/maskclip_utils/prompt_engineering.py"

    %% Click Events for Model Building & Training
    click TR "https://github.com/vobecant/pop3d/blob/main/train.py"
    click PC "https://github.com/vobecant/pop3d/blob/main/config/pop3d_maskclip_12ep.py"
    click CB1 "https://github.com/vobecant/pop3d/blob/main/builder/tpv_occupancy_builder.py"
    click CB2 "https://github.com/vobecant/pop3d/blob/main/builder/tpv_lidarseg_builder.py"
    click TP "https://github.com/vobecant/pop3d/tree/main/tpvformer04/"
    click CP "https://github.com/vobecant/pop3d/blob/main/config/_base_/optimizer.py"

    %% Click Events for Evaluation & Retrieval
    click EV "https://github.com/vobecant/pop3d/blob/main/eval.py"
    click RET "https://github.com/vobecant/pop3d/blob/main/retrieval.py"
    click SLURM_EV "https://github.com/vobecant/pop3d/blob/main/scripts/eval_zeroshot_slurm.sh"

    %% Click Events for Utilities & Integration
    click DL "https://github.com/vobecant/pop3d/tree/main/dataloader/"
    click VIS "https://github.com/vobecant/pop3d/tree/main/visualization/"
    click UT "https://github.com/vobecant/pop3d/tree/main/utils/"
    click DTS1 "https://github.com/vobecant/pop3d/blob/main/scripts/train_slurm.sh"
    click DTS2 "https://github.com/vobecant/pop3d/blob/main/scripts/train_pbs.sh"
    click DTS3 "https://github.com/vobecant/pop3d/blob/main/scripts/launcher.sh"
    click Loss "https://github.com/vobecant/pop3d/blob/main/builder/loss_builder.py"

    %% Styles
    classDef dataset fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef script fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef config fill:#A9DFBF,stroke:#229954,stroke-width:2px;
    classDef maskclip fill:#F5CBA7,stroke:#E67E22,stroke-width:2px;
    classDef trainer fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef builder fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef evaluation fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef utility fill:#D6EAF8,stroke:#2471A3,stroke-width:2px;
    classDef env fill:#D5F5E3,stroke:#28B463,stroke-width:2px;
```
