How facebookresearch/locate-3d works
Project Type and Purpose
• This is an open-source machine learning research codebase (Locate-3D) for 3D object localization via self-supervised learning.
• It bundles three major pieces: the Locate-3D Dataset (data loaders, utilities), the 3D-JEPA self-supervised encoder, and the language-conditioned decoder for object localization.
File Structure Analysis
• Top-level directories:
– assets: visualization images
– config: experiment configurations (YAML)
– examples: runnable scripts/notebooks showing usage
– locate3d_data: dataset definitions, data loaders, dataset files
– models: model classes (encoder, decoder, utilities)
– preprocessing: data/feature preprocessing pipeline
– tests: unit tests
• Patterns:
– ML pipeline separation: data ingestion (locate3d_data), preprocessing (image & pointcloud featurization), model definitions (models), experiment config (config), entry-points (examples).
– Utilities subpackages under models and preprocessing for modularity.
README Insights
• Describes three core subsystems: Locate-3D Dataset, 3D-JEPA encoder, Locate-3D model (encoder + language decoder).
• Notes dependencies on 2D foundation models (CLIP, DINO, SAM) for feature extraction.
• Points to example scripts and configuration files for running experiments.
a. Main Components
1. Data Sources
– Raw Sensor Streams (posed RGB-D frames)
– JSON annotations (Locate-3D Dataset)
2. Preprocessing Pipeline
– Image feature extraction (CLIP, DINO, SAM encoders)
– Pointcloud featurizer & voxelization
– RGB-D unprojection and SLAM transforms
3. Model Core
– 3D-JEPA Encoder (masked latent prediction)
4. Training & Evaluation
– Config loader (YAML)
– Training scripts (examples/run_*.py)
– Metrics & visualization utilities
5. External Dependencies
– PyTorch, Hydra (config), Hugging Face models, SLURM (for cluster jobs)
b. Relationships & Data Flow
1. Raw data → preprocessing → feature tensors
2. Feature tensors → 3D-JEPA Encoder → contextualized 3D features
4. Predictions → evaluation metrics & visualization
5. Config files drive hyperparameters and dataset splits
c. Architectural Patterns
• Modular pipeline with clear separation of concerns: data, preprocessing, model, config, examples.
• Self-supervised pretraining (3D-JEPA) followed by supervised finetuning (Locate-3D).
• Plugin-style feature extractors: each 2D encoder (CLIP, DINO, SAM) lives under preprocessing/image_features.
• Hydra-style configuration for reproducible experiments.
d. Key Technologies & Libraries
• Deep Learning: PyTorch
• Configuration: Hydra / YAML
• Foundation Models: CLIP, DINO, SAM via Hugging Face
• SLAM utilities for RGB-D (preprocessing)
• Dataset handling: custom PyTorch Dataset classes