How facebookresearch/uco3d works
Project Type and Purpose
• The repository is primarily a dataset management and processing toolkit for the uCO3D dataset (“UnCommon Objects in 3D”). Its main functionalities include downloading, organizing, and interfacing with a large-scale 3D video dataset.
• It provides both the means to download the full dataset (or subsets, specific modalities, or super-categories) and an API to load, access, and process the 3D scene data (e.g., reading videos, loading h5 depth maps, handling point clouds and 3D Gaussian splats).
• Additional support is provided for conversion utilities (e.g., converting camera formats to PyTorch3D or OpenCV conventions) and visualization (via integrated rendering using the gsplat library).
Key Components Identified from File Structure and README
A. Download Manager:
• Found under the dataset_download directory – includes scripts (download_dataset.py and its implementation) and related configuration (e.g., category_to_archives.json) to manage data download and decompression.
• This component handles modality-specific and category-specific downloading.
B. Core API and Data Processing Library:
• The uco3d package forms the core library providing the API. Key modules are:
– UCO3DDataset: Initiates dataset access leveraging the local file system and the SQLite metadata.
– UCO3DFrameDataBuilder: Processes per-frame data by extracting images, depths, masks, Gaussian splats, and point clouds from the downloaded videos and h5 files.
• The dataset_utils submodule (within uco3d/dataset_utils) clusters functions related to dataset annotations, type definitions, file I/O, and conversion utilities (e.g., converting annotations or rendering Gaussian splats).
C. Conversion and Rendering Modules:
• Helper functions are provided for converting between different camera representations (e.g., opencv_cameras_projection_from_uco3d) and rendering (render_splats using a gsplat dependency).
• These facilitate interactions with external frameworks (PyTorch3D, OpenCV) and external libraries like gsplat.
D. Examples and Testing:
• The examples folder includes scripts that demonstrate real-world usage (e.g., rendering Gaussian turntables).
• The tests folder verifies the correctness of utilities and visualizations with multiple scripts testing cameras, dataloaders, Gaussian splats, and more.
E. External Data and Infrastructure:
• The downloaded dataset on disk is organized into a hierarchical file system (by super-category, category, and sequence). This structure is central to the design since it influences how the API retrieves and processes data.
• There is also a metadata database (metadata.sqlite) storing frame-level and video-level annotations, and a set of SQLite “set_lists” files managing dataset splits.
Relationships and Interactions
• The Download Manager interacts with the local file system to retrieve and store videos, h5 files, and supplementary data. This downloaded content follows a well-organized hierarchy.
• The Core API (via UCO3DDataset and UCO3DFrameDataBuilder) reads from the file system and the SQLite metadata database to “assemble” a scene’s data (including selecting a frame, loading camera parameters, images, and 3D reconstructions).
• The conversion utilities in dataset_utils communicate with external libraries (PyTorch3D and OpenCV) to offer consistent camera representations.
• The rendering module (render_splats) serves as the bridge between raw 3D data and visual output, requiring the gsplat library for processing Gaussian splats.
• Examples and tests serve as usage validation layers that interact with nearly all above components.
Architectural Patterns and Important Details
• Modular Design: Components are distinctly separated (download, data processing, conversion, rendering, testing) enabling independent evolution and testing.
• API Layer: The uco3d package provides a high-level object-oriented API that abstracts away the complexities of the file system layout and data formats.
• Data Flow: Emphasize that configuration (e.g., UCO3D_DATASET_ROOT) guides access to both the file system and metadata database. This configuration feeds into the dataset builder, which then “pipes” data onward to conversion and rendering modules.
Final Remarks
• The goal is to capture the project’s modular nature along with its interactions with external systems and dependencies.