# apple/ml-cubifyanything architecture

> Architecture diagram of the GitHub repository apple/ml-cubifyanything, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-cubifyanything
- Repository: https://github.com/apple/ml-cubifyanything

## Overview

##### Project type and purpose
- This is an open-source Python library and CLI demo tool for 3D object detection “Cubify Transformer” (CuTR) built around the CA-1M indoor scene dataset.
- It provides dataset readers (WebDataset), preprocessing, model loading/inference (RGB and RGB-D variants), optional live stream via NeRF Capture, and 3D visualization via rerun.

##### File structure
- Top-level metadata (LICENSE, README.md, requirements.txt, setup.py) indicates a pip-installable package.
- cubifyanything/ – core library modules:
– dataset.py, imagelist.py, batching.py, preprocessor.py, transforms.py – data ingestion & preprocessing pipeline
– cubify_transformer.py, vit.py – model definition and Transformer backbone
– sensor.py, capture_stream.py – live stream capture support
- data/ – pointers to download CA-1M WebDataset archives (train.txt, val.txt) and data license
- tools/demo.py – CLI entrypoint orchestrating data loading, inference, and visualization

##### README insights
- Data in WebDataset .tar per video, images, depth, poses, 3D annotations
- Two model variants (RGB, RGB-D) loaded via CLI arguments
- Visualization handled by rerun, optional streaming input from NeRF Capture
- External dependencies: PyTorch, torchvision, rerun

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Source"
        direction TB
        WebDataset[("WebDataset archives")]:::data
        CaptureStream[("NeRF Capture\n(capture_stream.py)")]:::data
        SensorModule[("Sensor module\n(sensor.py)")]:::data
    end

    %% Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        Dataset["CubifyAnythingDataset\n(dataset.py)"]:::processing
        ImageList["ImageList utility\n(imagelist.py)"]:::processing
        Batching["Batching pipeline\n(batching.py)"]:::processing
        Preprocessor["Preprocessor\n(preprocessor.py)"]:::processing
        Transforms["Transforms definitions\n(transforms.py)"]:::processing
    end

    %% Model Inference
    subgraph "Model Inference"
        direction TB
        CubifyModel["Cubify Transformer\n(cubify_transformer.py)"]:::model
        ViT["ViT Backbone\n(vit.py)"]:::model
        ModelArtifacts[("Model Artifacts\n(local cache & remote .pth)")]:::external
    end

    %% Post-Processing
    subgraph "Post-Processing"
        direction TB
        Boxes["3D Box Fitting\n(boxes.py)"]:::post
        Instances["Instance Aggregation\n(instances.py)"]:::post
        Measurement["Measurement & Metrics\n(measurement.py)"]:::post
        Orientation["Orientation Processing\n(orientation.py)"]:::post
        Position["Position Handling\n(pos.py)"]:::post
        Color["Color Info\n(color.py)"]:::post
    end

    %% CLI & Visualization
    subgraph "CLI Orchestration & Visualization"
        direction TB
        Demo["Demo CLI\n(tools/demo.py)"]:::orchestrator
        Rerun["3D Visualization\n(rerun)"]:::visualize
    end

    %% Package Metadata
    Requirements[("Package Metadata\n(requirements.txt, setup.py)")]:::metadata

    %% Data Flow
    WebDataset --> Dataset
    CaptureStream --> Dataset
    SensorModule --> Dataset

    Dataset --> ImageList --> Batching --> Preprocessor --> Transforms --> CubifyModel
    ViT --> CubifyModel
    ModelArtifacts --> CubifyModel

    CubifyModel --> Boxes --> Instances --> Measurement --> Orientation --> Position --> Color --> Rerun

    %% Orchestration
    Demo --> Dataset
    Demo --> CubifyModel
    Demo --> Boxes
    Demo --> Rerun

    %% Dependencies
    Requirements -.-> Demo
    Requirements -.-> Dataset
    Requirements -.-> CubifyModel
    Requirements -.-> Rerun

    %% Legend (using dummy nodes)
    subgraph Legend[ ]
        direction TB
        A["Blue = Data"]:::data
        B["Green = Processing"]:::processing
        C["Purple = Model"]:::model
        D["Orange = Post-Process"]:::post
        E["Red = Visualization"]:::visualize
        F["Gray = Metadata"]:::metadata
    end

    %% Click Events
    click WebDataset "https://github.com/apple/ml-cubifyanything/blob/main/data/train.txt"
    click CaptureStream "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/capture_stream.py"
    click SensorModule "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/sensor.py"
    click Dataset "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/dataset.py"
    click ImageList "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/imagelist.py"
    click Batching "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/batching.py"
    click Preprocessor "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/preprocessor.py"
    click Transforms "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/transforms.py"
    click CubifyModel "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/cubify_transformer.py"
    click ViT "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/vit.py"
    click Boxes "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/boxes.py"
    click Instances "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/instances.py"
    click Measurement "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/measurement.py"
    click Orientation "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/orientation.py"
    click Position "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/pos.py"
    click Color "https://github.com/apple/ml-cubifyanything/blob/main/cubifyanything/color.py"
    click Demo "https://github.com/apple/ml-cubifyanything/blob/main/tools/demo.py"
    click Requirements "https://github.com/apple/ml-cubifyanything/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#D0E6FA,stroke:#4A90E2,color:#1A3E6D;
    classDef processing fill:#E8F7E4,stroke:#34A853,color:#0F5628;
    classDef model fill:#F3E8FD,stroke:#A142F4,color:#581B7E;
    classDef post fill:#FFEAD2,stroke:#F2994A,color:#7A4900;
    classDef visualize fill:#FDECEA,stroke:#EB5757,color:#7A1A1A;
    classDef orchestrator fill:#E8F0FE,stroke:#2F80ED,color:#1230A0;
    classDef external fill:#F0F0F0,stroke:#CCCCCC,color:#555555,stroke-dasharray: 5 5;
    classDef metadata fill:#F4F4F4,stroke:#999999,color:#333333,stroke-dasharray: 2 2;
```
