# facebookresearch/locate-3d architecture

> Architecture diagram of the GitHub repository facebookresearch/locate-3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/locate-3d
- Repository: https://github.com/facebookresearch/locate-3d

## Overview

##### Project Type and Purpose
- This is an open-source machine learning research codebase (Locate-3D) for 3D object localization via self-supervised learning.
- It bundles three major pieces: the Locate-3D Dataset (data loaders, utilities), the 3D-JEPA self-supervised encoder, and the language-conditioned decoder for object localization.

##### File Structure Analysis
- Top-level directories:
– assets: visualization images
– config: experiment configurations (YAML)
– examples: runnable scripts/notebooks showing usage
– locate3d_data: dataset definitions, data loaders, dataset files
– models: model classes (encoder, decoder, utilities)
– preprocessing: data/feature preprocessing pipeline
– tests: unit tests
- Patterns:
– ML pipeline separation: data ingestion (locate3d_data), preprocessing (image & pointcloud featurization), model definitions (models), experiment config (config), entry-points (examples).
– Utilities subpackages under models and preprocessing for modularity.

##### README Insights
- Describes three core subsystems: Locate-3D Dataset, 3D-JEPA encoder, Locate-3D model (encoder + language decoder).
- Notes dependencies on 2D foundation models (CLIP, DINO, SAM) for feature extraction.
- Points to example scripts and configuration files for running experiments.

a. Main Components
1. Data Sources
– Raw Sensor Streams (posed RGB-D frames)
– JSON annotations (Locate-3D Dataset)
2. Preprocessing Pipeline
– Image feature extraction (CLIP, DINO, SAM encoders)
– Pointcloud featurizer & voxelization
– RGB-D unprojection and SLAM transforms
3. Model Core
– 3D-JEPA Encoder (masked latent prediction)
4. Training & Evaluation
– Config loader (YAML)
– Training scripts (examples/run_*.py)
– Metrics & visualization utilities
5. External Dependencies
– PyTorch, Hydra (config), Hugging Face models, SLURM (for cluster jobs)

b. Relationships & Data Flow
1. Raw data → preprocessing → feature tensors
2. Feature tensors → 3D-JEPA Encoder → contextualized 3D features
4. Predictions → evaluation metrics & visualization
5. Config files drive hyperparameters and dataset splits

c. Architectural Patterns
- Modular pipeline with clear separation of concerns: data, preprocessing, model, config, examples.
- Self-supervised pretraining (3D-JEPA) followed by supervised finetuning (Locate-3D).
- Plugin-style feature extractors: each 2D encoder (CLIP, DINO, SAM) lives under preprocessing/image_features.
- Hydra-style configuration for reproducible experiments.

d. Key Technologies & Libraries
- Deep Learning: PyTorch
- Configuration: Hydra / YAML
- Foundation Models: CLIP, DINO, SAM via Hugging Face
- SLAM utilities for RGB-D (preprocessing)
- Dataset handling: custom PyTorch Dataset classes

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        direction TB
        Raw["Raw Sensor Streams"]:::data
        JSON["JSON Annotations"]:::data
        subgraph "Locate-3D Dataset"
            direction TB
            L3D["locate3d_data/locate3d_dataset.py"]:::data
            ARKIT["arkitscenes_dataset.py"]:::data
            SCAN["scannet_dataset.py"]:::data
            SCANPP["scannetpp_dataset.py"]:::data
            DUTIL["data_utils.py"]:::data
            VUTIL["vis_utils.py"]:::data
            JSONS["JSON Files"]:::data
        end
    end

    subgraph "Preprocessing Layer"
        direction TB
        subgraph "Image Features"
            direction TB
            CLIP["clip_encoder.py"]:::prep
            DINO["dino_encoder.py"]:::prep
            SAM["sam_encoder.py"]:::prep
            IEMB["image_embedding.py"]:::prep
            IFUTIL["image_feature_utils.py"]:::prep
            MEB["mask_embedding.py"]:::prep
        end
        subgraph "Pointcloud & SLAM"
            direction TB
            PCF["pointcloud_featurizer.py"]:::prep
            VOX["voxelized_pointcloud.py"]:::prep
            SLAMTRANS["feature_slam_transform.py"]:::prep
            UNPROJ["unproject_rgbd_feature_slam.py"]:::prep
            RUNPRE["run_preprocessing.py"]:::prep
            RUNPRESL["run_preprocessing_slurm_array.py"]:::prep
        end
        FT["Feature Tensors"]:::prep
    end

    subgraph "Model Layer"
        direction TB
        Encoder3D["encoder_3djepa.py"]:::model
        CF["Contextualized 3D Features"]:::model
        subgraph "Locate-3D Decoder"
            direction TB
            DEC["locate_3d_decoder.py"]:::model
            LOC["locate_3d.py"]:::model
        end
        subgraph "Model Utilities"
            direction TB
            BBOX["bbox_utils.py"]:::model
            SERDEF["default.py"]:::model
            SERHIL["hilbert.py"]:::model
            SERZ["z_order.py"]:::model
        end
    end

    subgraph "Config & Training"
        direction TB
        JEPA_CFG["jepa_3d.yaml"]:::config
        LOC_CFG["locate_3d.yaml"]:::config
        EX3D["3d_jepa_ex.py"]:::config
        EXLOC["locate_3d_ex.py"]:::config
        EXPLUS["locate_3d_plus_ex.py"]:::config
        IPY["dataset_playground.ipynb"]:::config
    end

    subgraph "Evaluation & Visualization"
        direction TB
        Metrics["Metrics & Visualization Utilities"]:::eval
        Assets["assets/"]:::eval
    end

    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch"]:::ext
        Hydra["Hydra"]:::ext
        HF["Hugging Face Models"]:::ext
        SLURM["SLURM"]:::ext
        TE["Text Embeddings"]:::ext
    end

    Raw --> L3D
    JSON --> L3D
    L3D --> FT
    CLIP --> FT
    DINO --> FT
    SAM --> FT
    IEMB --> FT
    IFUTIL --> FT
    MEB --> FT
    PCF --> FT
    VOX --> FT
    SLAMTRANS --> FT
    UNPROJ --> FT
    FT --> Encoder3D
    Encoder3D --> CF
    HF --> TE
    TE --> DEC
    CF --> DEC
    DEC --> Pred["3D Masks & Bounding Boxes"]:::model
    Pred --> Metrics
    Metrics --> Assets

    JEPA_CFG --> EX3D
    LOC_CFG --> EXLOC
    LOC_CFG --> EXPLUS
    RUNPRESL --> SLURM

    classDef data fill:#cce5ff,stroke:#004085;
    classDef prep fill:#d4edda,stroke:#155724;
    classDef model fill:#e2dfff,stroke:#4b0082;
    classDef config fill:#ffe5b4,stroke:#cc8400;
    classDef eval fill:#f8f9fa,stroke:#6c757d;
    classDef ext fill:#e2e3e5,stroke:#6c757d;

    click L3D "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/locate3d_dataset.py"
    click ARKIT "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/arkitscenes_dataset.py"
    click SCAN "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/scannet_dataset.py"
    click SCANPP "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/scannetpp_dataset.py"
    click DUTIL "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/data_utils.py"
    click VUTIL "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/vis_utils.py"
    click JSONS "https://github.com/facebookresearch/locate-3d/blob/main/locate3d_data/dataset/*.json"
    click CLIP "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/clip_encoder.py"
    click DINO "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/dino_encoder.py"
    click SAM "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/sam_encoder.py"
    click IEMB "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/image_embedding.py"
    click IFUTIL "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/image_feature_utils.py"
    click MEB "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/image_features/mask_embedding.py"
    click PCF "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/pointcloud_featurizer.py"
    click VOX "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/voxelized_pointcloud.py"
    click SLAMTRANS "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/feature_slam_transform.py"
    click UNPROJ "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/unproject_rgbd_feature_slam.py"
    click RUNPRE "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/run_preprocessing.py"
    click RUNPRESL "https://github.com/facebookresearch/locate-3d/blob/main/preprocessing/run_preprocessing_slurm_array.py"
    click Encoder3D "https://github.com/facebookresearch/locate-3d/blob/main/models/encoder_3djepa.py"
    click DEC "https://github.com/facebookresearch/locate-3d/blob/main/models/locate_3d_decoder.py"
    click LOC "https://github.com/facebookresearch/locate-3d/blob/main/models/locate_3d.py"
    click BBOX "https://github.com/facebookresearch/locate-3d/blob/main/models/model_utils/bbox_utils.py"
    click SERDEF "https://github.com/facebookresearch/locate-3d/blob/main/models/model_utils/serialization/default.py"
    click SERHIL "https://github.com/facebookresearch/locate-3d/blob/main/models/model_utils/serialization/hilbert.py"
    click SERZ "https://github.com/facebookresearch/locate-3d/blob/main/models/model_utils/serialization/z_order.py"
    click JEPA_CFG "https://github.com/facebookresearch/locate-3d/blob/main/config/jepa_3d.yaml"
    click LOC_CFG "https://github.com/facebookresearch/locate-3d/blob/main/config/locate_3d.yaml"
    click EX3D "https://github.com/facebookresearch/locate-3d/blob/main/examples/3d_jepa_ex.py"
    click EXLOC "https://github.com/facebookresearch/locate-3d/blob/main/examples/locate_3d_ex.py"
    click EXPLUS "https://github.com/facebookresearch/locate-3d/blob/main/examples/locate_3d_plus_ex.py"
    click IPY "https://github.com/facebookresearch/locate-3d/blob/main/examples/dataset_playground.ipynb"
    click Assets "https://github.com/facebookresearch/locate-3d/tree/main/assets/"
```
