# facebookresearch/uco3d architecture

> Architecture diagram of the GitHub repository facebookresearch/uco3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/uco3d
- Repository: https://github.com/facebookresearch/uco3d

## Overview

##### Project Type and Purpose
- The repository is primarily a dataset management and processing toolkit for the uCO3D dataset (“UnCommon Objects in 3D”). Its main functionalities include downloading, organizing, and interfacing with a large-scale 3D video dataset.
- It provides both the means to download the full dataset (or subsets, specific modalities, or super-categories) and an API to load, access, and process the 3D scene data (e.g., reading videos, loading h5 depth maps, handling point clouds and 3D Gaussian splats).
- Additional support is provided for conversion utilities (e.g., converting camera formats to PyTorch3D or OpenCV conventions) and visualization (via integrated rendering using the gsplat library).

##### Key Components Identified from File Structure and README
A. Download Manager:
- Found under the dataset_download directory – includes scripts (download_dataset.py and its implementation) and related configuration (e.g., category_to_archives.json) to manage data download and decompression.
- This component handles modality-specific and category-specific downloading.

B. Core API and Data Processing Library:
- The uco3d package forms the core library providing the API. Key modules are:
– UCO3DDataset: Initiates dataset access leveraging the local file system and the SQLite metadata.
– UCO3DFrameDataBuilder: Processes per-frame data by extracting images, depths, masks, Gaussian splats, and point clouds from the downloaded videos and h5 files.
- The dataset_utils submodule (within uco3d/dataset_utils) clusters functions related to dataset annotations, type definitions, file I/O, and conversion utilities (e.g., converting annotations or rendering Gaussian splats).

C. Conversion and Rendering Modules:
- Helper functions are provided for converting between different camera representations (e.g., opencv_cameras_projection_from_uco3d) and rendering (render_splats using a gsplat dependency).
- These facilitate interactions with external frameworks (PyTorch3D, OpenCV) and external libraries like gsplat.

D. Examples and Testing:
- The examples folder includes scripts that demonstrate real-world usage (e.g., rendering Gaussian turntables).
- The tests folder verifies the correctness of utilities and visualizations with multiple scripts testing cameras, dataloaders, Gaussian splats, and more.

E. External Data and Infrastructure:
- The downloaded dataset on disk is organized into a hierarchical file system (by super-category, category, and sequence). This structure is central to the design since it influences how the API retrieves and processes data.
- There is also a metadata database (metadata.sqlite) storing frame-level and video-level annotations, and a set of SQLite “set_lists” files managing dataset splits.

##### Relationships and Interactions
- The Download Manager interacts with the local file system to retrieve and store videos, h5 files, and supplementary data. This downloaded content follows a well-organized hierarchy.
- The Core API (via UCO3DDataset and UCO3DFrameDataBuilder) reads from the file system and the SQLite metadata database to “assemble” a scene’s data (including selecting a frame, loading camera parameters, images, and 3D reconstructions).
- The conversion utilities in dataset_utils communicate with external libraries (PyTorch3D and OpenCV) to offer consistent camera representations.
- The rendering module (render_splats) serves as the bridge between raw 3D data and visual output, requiring the gsplat library for processing Gaussian splats.
- Examples and tests serve as usage validation layers that interact with nearly all above components.

##### Architectural Patterns and Important Details
- Modular Design: Components are distinctly separated (download, data processing, conversion, rendering, testing) enabling independent evolution and testing.
- API Layer: The uco3d package provides a high-level object-oriented API that abstracts away the complexities of the file system layout and data formats.
- Data Flow: Emphasize that configuration (e.g., UCO3D_DATASET_ROOT) guides access to both the file system and metadata database. This configuration feeds into the dataset builder, which then “pipes” data onward to conversion and rendering modules.

##### Final Remarks
- The goal is to capture the project’s modular nature along with its interactions with external systems and dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% A. Download Manager Subgraph
    subgraph "Download Manager"
        DM1["download_dataset.py"]:::download
        DM2["download_dataset_impl.py"]:::download
        DM3["category_to_archives.json"]:::download
    end

    %% B. Data Storage Subgraph
    subgraph "Data Storage"
        DS1["File System Hierarchy"]:::datastore
        DS2["Metadata Database (metadata.sqlite & set_lists)"]:::datastore
    end

    %% C. Core API Subgraph
    subgraph "Core API"
        API1["UCO3DDataset"]:::api
        API2["UCO3DFrameDataBuilder"]:::api
    end

    %% D. Utilities & Modules Subgraph
    subgraph "Utilities & Modules"
        DU["Dataset Utilities"]:::utils
        CM["Conversion Module"]:::conversion
        RM["Rendering Module"]:::rendering
    end

    %% E. External Libraries Subgraph
    subgraph "External"
        EX1["OpenCV & PyTorch3D"]:::external
        EX2["gsplat"]:::external
    end

    %% F. Usage Layer Subgraph
    subgraph "Usage"
        EXM["Examples"]:::usage
        TEST["Testing"]:::usage
    end

    %% Connections from Download Manager to Data Storage
    DM1 -->|"download_and_store"| DS1
    DM2 -->|"download_and_store"| DS1
    DM3 -->|"download_and_store"| DS1

    %% Connections from Data Storage to Core API
    DS1 -->|"read_files"| API1
    DS1 -->|"read_files"| API2
    DS2 -->|"parse_metadata"| API1
    DS2 -->|"parse_metadata"| API2

    %% Connections from Core API to Utilities & Modules
    API1 -->|"process_data"| DU
    API2 -->|"process_data"| DU

    %% Connections from Dataset Utilities to Conversion and Rendering Modules
    DU -->|"conversion"| CM
    DU -->|"rendering"| RM

    %% Connections from Conversion and Rendering Modules to External Libraries
    CM -->|"convert_camera_format"| EX1
    RM -->|"render_scene"| EX2

    %% Connections from Core API to Usage layer
    API1 -->|"usage"| EXM
    API2 -->|"usage"| TEST

    %% Styles
    classDef download fill:#FFCC00,stroke:#000,stroke-width:2px;
    classDef datastore fill:#F0E68C,stroke:#000,stroke-width:2px;
    classDef api fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef utils fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef conversion fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef rendering fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef external fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef usage fill:#C0C0C0,stroke:#000,stroke-width:2px;

    %% Click Events
    click DM1 "https://github.com/facebookresearch/uco3d/blob/main/dataset_download/download_dataset.py"
    click DM2 "https://github.com/facebookresearch/uco3d/blob/main/dataset_download/download_dataset_impl.py"
    click DM3 "https://github.com/facebookresearch/uco3d/blob/main/dataset_download/category_to_archives.json"
    click API1 "https://github.com/facebookresearch/uco3d/blob/main/uco3d/uco3d_dataset.py"
    click API2 "https://github.com/facebookresearch/uco3d/blob/main/uco3d/uco3d_frame_data_builder.py"
    click DU "https://github.com/facebookresearch/uco3d/tree/main/uco3d/dataset_utils/"
    click CM "https://github.com/facebookresearch/uco3d/blob/main/uco3d/dataset_utils/gauss3d_convert.py"
    click RM "https://github.com/facebookresearch/uco3d/blob/main/uco3d/dataset_utils/gauss3d_rendering.py"
    click EXM "https://github.com/facebookresearch/uco3d/tree/main/examples/"
    click TEST "https://github.com/facebookresearch/uco3d/tree/main/tests/"
```
