How mako443/text2pos-cvpr2022 works
Project Type and Purpose
• This project is a research prototype aimed at “Text-to-point-cloud cross-modal localization.” It implements a coarse-to-fine localization strategy to align natural language descriptions with point-cloud localization cues.
• The project’s purpose is to process and prepare a specialized dataset (KITTI360Pose derived from the KITTI360 dataset), train models (coarse and fine localization networks), and evaluate the localization performance.
Break Down the File Structure into Main Components
• Data Preparation:
• The “datapreparation” directory is dedicated to processing raw KITTI360 data to create the KITTI360Pose dataset. Files like prepare.py, prepare_images.py, and rendering/drawing scripts highlight the key steps in data transformation.
• Data Loading:
• The “dataloading” folder, with its subdirectory for KITTI360Pose, contains modules for loading the different aspects of the dataset (poses, images, cells, objects, etc.).
• Model Definitions:
• The “models” directory contains architecture definitions for components such as cell retrieval, offset regression, object encoding, point cloud processing (with modules like pointnet2), and transformer-based parts, including modules for SuperGlue and matcher components.
• Training Modules:
• The “training” folder holds training routines for both coarse and fine networks, including training scripts (coarse.py, fine.py), losses, and even training versions of point-cloud modules.
• Evaluation Pipeline:
• The “evaluation” directory includes pipeline scripts and evaluation utilities that combine model inference with visual localization outputs.
• Utilities and Scripts:
• The “plots” directory is used for visualization.
• The “slurm” directory contains shell scripts for automated runs (training and evaluation) on clusters.
• Support Files:
• README.md explains installation, dependency management, and how to run evaluation/training.
• Dependencies are specified in requirements.txt, and there are checkpoint files with pre-trained models stored in the “checkpoints” directory.
• Data Source:
• External KITTI360 dataset and generated KITTI360Pose dataset.
• Data Preparation Module:
• Represented by the datapreparation component (includes modules for data transformation, image preparation, pose rendering, etc.).
• Data Loading Module:
• Represented by the dataloading layer, handling various KITTI360Pose aspects such as images, poses, and objects.
• Model Components:
• Coarse localization network (likely involving cell retrieval and initial pose estimation).
• Fine localization network (with components such as transformer-based modules and offset regression).
• Additional model blocks such as pointcloud (pointnet2) and matching networks (SuperGlue, TF matcher).
• Training Mechanism:
• Training scripts (from the training directory) that orchestrate the training of both coarse and fine models.
• Evaluation Pipeline:
• The evaluation component that loads checkpoints and runs localization evaluation to generate quantitative and visual outputs.
• Deployment/Execution Infrastructure:
• Slurm scripts indicating deployment on compute clusters.
• Visualization Tools:
• Plot generation routines and optional packages like pptk for point-cloud visualization.
(b) Define Relationships and Data Flow:
• Data flows from the raw KITTI360 dataset into datapreparation. The prepared data, stored in a designated folder (e.g., “./data/k360_30-10_scG_pd10_pc4_spY_all_nm6/”), is then loaded via the dataloading modules.
• The training modules (for coarse and fine models) receive data from the dataloading layer and use model definitions from the models folder to train localization networks.
• Model checkpoints (stored in the “checkpoints” directory) are later used in the evaluation pipeline. The evaluation process also involves visualizations via the plots module.
• The slurm scripts encapsulate commands that trigger either training or evaluation, indicating the system’s execution environment (batch jobs on clusters).
Architectural Patterns and Design Principles to Highlight
• Separation of Concerns: Notice how data preparation, data loading, modeling, training, and evaluation are distinct modules, each encapsulated in its own directory.
• Modularity: The models are broken down into submodules focusing on various components (e.g., point cloud processing, matching, transformer modules) which lends itself to reusability and extensibility.