How facebookresearch/vggt-omega works
VGGT-Ω is a multi-view geometry model: given images or video frames, it predicts camera pose and dense depth, with optional text-aligned embeddings; the demo turns these predictions into a visualized 3D scene. The repository also contains training, dataset curation, and benchmark evaluation workflows. The inference path is documented in the README and demo imports; training internals are only partly excerpted, so uncertain connections are omitted or marked unverified.
Main entry points: vggt_omega.py VGGT-Ω Model · demo_gradio.py Gradio Demo · trainer.py Training Orchestrator · load_fn.py Image Preprocessing
Components 20
Inference Model
- Image Preprocessing
vggt_omega/utils/load_fn.py - VGGT-Ω Model
vggt_omega/models/vggt_omega.py - Frame Aggregator
vggt_omega/models/aggregator.py - Vision Transformer
vggt_omega/models/layers/vision_transformer.py
Geometry Outputs
- Camera Head
vggt_omega/models/heads/camera_head.py - Dense Prediction Head
vggt_omega/models/heads/dense_head.py - Text Alignment Head
vggt_omega/models/heads/text_alignment_head.py - Pose Decoder
vggt_omega/utils/pose_enc.py - Rotation Utilities
vggt_omega/utils/rotation.py
Interactive Demo
- Gradio Demo
demo_gradio.py - 3D Visualization
visual_util.py
Model Training
- Training Orchestrator
training/trainer.py - Unified Datasets
training/data/datasets/unified.py - Dataset Utilities
training/data/base_dataset.py - Training Losses
training/losses/loss.py - Checkpoint Handling
training/train_utils/checkpoint.py
Data Curation & Evaluation
- Geometric Curation Metrics
training/curation/compute_metrics.py - Benchmark Evaluation
eval/evaluate.py - Evaluation Metrics
eval/metrics.py
Other components
- Image / Video User
Connections 20
- Image / Video User → Image Preprocessing: provides images
README.md - Image Preprocessing → VGGT-Ω Model: prepares tensors
README.md - VGGT-Ω Model → Frame Aggregator: encodes frames
- Frame Aggregator → Vision Transformer: uses blocks
vggt_omega/models/aggregator.py - VGGT-Ω Model → Camera Head: predicts cameras
- VGGT-Ω Model → Dense Prediction Head: predicts depth
- VGGT-Ω Model → Text Alignment Head: aligns text
- Camera Head → Pose Decoder: encodes pose
- Pose Decoder → Rotation Utilities: converts rotations
vggt_omega/utils/pose_enc.py - Image / Video User → Gradio Demo: uploads media
README.md - Gradio Demo → Image Preprocessing: loads images
demo_gradio.py - Gradio Demo → VGGT-Ω Model: runs inference
README.md - Gradio Demo → Pose Decoder: decodes cameras
demo_gradio.py - Gradio Demo → 3D Visualization: renders scene
demo_gradio.py - Training Orchestrator → Unified Datasets: instantiates dataset
training/trainer.py - Unified Datasets → Dataset Utilities: uses base dataset
training/data/datasets/unified.py - Training Orchestrator → Training Losses: computes objective
- Training Orchestrator → Checkpoint Handling: saves state
training/trainer.py - Geometric Curation Metrics → Unified Datasets: reads sequences
- Benchmark Evaluation → Evaluation Metrics: computes scores