How apple/ml-cubifyanything works
Project type and purpose
• This is an open-source Python library and CLI demo tool for 3D object detection “Cubify Transformer” (CuTR) built around the CA-1M indoor scene dataset.
• It provides dataset readers (WebDataset), preprocessing, model loading/inference (RGB and RGB-D variants), optional live stream via NeRF Capture, and 3D visualization via rerun.
File structure
• Top-level metadata (LICENSE, README.md, requirements.txt, setup.py) indicates a pip-installable package.
• cubifyanything/ – core library modules:
– dataset.py, imagelist.py, batching.py, preprocessor.py, transforms.py – data ingestion & preprocessing pipeline
– cubify_transformer.py, vit.py – model definition and Transformer backbone
– sensor.py, capture_stream.py – live stream capture support
• data/ – pointers to download CA-1M WebDataset archives (train.txt, val.txt) and data license
• tools/demo.py – CLI entrypoint orchestrating data loading, inference, and visualization
README insights
• Data in WebDataset .tar per video, images, depth, poses, 3D annotations
• Two model variants (RGB, RGB-D) loaded via CLI arguments
• Visualization handled by rerun, optional streaming input from NeRF Capture
• External dependencies: PyTorch, torchvision, rerun