# facebookresearch/vggt-omega architecture

> Architecture diagram of the GitHub repository facebookresearch/vggt-omega, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/vggt-omega
- Repository: https://github.com/facebookresearch/vggt-omega

## Overview

VGGT-Ω is a multi-view geometry model: given images or video frames, it predicts camera pose and dense depth, with optional text-aligned embeddings; the demo turns these predictions into a visualized 3D scene. The repository also contains training, dataset curation, and benchmark evaluation workflows. The inference path is documented in the README and demo imports; training internals are only partly excerpted, so uncertain connections are omitted or marked unverified.

Main entry points: [`vggt_omega/models/vggt_omega.py`](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/vggt_omega.py) (VGGT-Ω Model), [`demo_gradio.py`](https://github.com/facebookresearch/vggt-omega/blob/main/demo_gradio.py) (Gradio Demo), [`training/trainer.py`](https://github.com/facebookresearch/vggt-omega/blob/main/training/trainer.py) (Training Orchestrator), [`vggt_omega/utils/load_fn.py`](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/load_fn.py) (Image Preprocessing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference Model"]
  node_image_prep["Image Preprocessing<br/>[load_fn.py]"]
  node_model["VGGT-Ω Model<br/>[vggt_omega.py]"]
  node_aggregator["Frame Aggregator<br/>[aggregator.py]"]
  node_transformer["Vision Transformer"]
end

subgraph group_geometry["Geometry Outputs"]
  node_camera_head["Camera Head<br/>[camera_head.py]"]
  node_dense_head["Dense Prediction Head<br/>[dense_head.py]"]
  node_text_head["Text Alignment Head"]
  node_pose_decode["Pose Decoder<br/>[pose_enc.py]"]
  node_rotation["Rotation Utilities<br/>[rotation.py]"]
end

subgraph group_demo["Interactive Demo"]
  node_demo_app["Gradio Demo<br/>[demo_gradio.py]"]
  node_visualization["3D Visualization<br/>[visual_util.py]"]
end

subgraph group_training["Model Training"]
  node_trainer["Training Orchestrator<br/>[trainer.py]"]
  node_datasets["Unified Datasets<br/>[unified.py]"]
  node_dataset_base["Dataset Utilities<br/>[base_dataset.py]"]
  node_losses["Training Losses<br/>[loss.py]"]
  node_checkpoint["Checkpoint Handling<br/>[checkpoint.py]"]
end

subgraph group_data_quality["Data Curation &amp; Evaluation"]
  node_curation["Geometric Curation Metrics<br/>[compute_metrics.py]"]
  node_eval["Benchmark Evaluation<br/>[evaluate.py]"]
  node_eval_metrics["Evaluation Metrics<br/>[metrics.py]"]
end

node_user(("Image / Video User"))

node_user -->|"provides images"| node_image_prep
node_image_prep -->|"prepares tensors"| node_model
node_model -.->|"encodes frames"| node_aggregator
node_aggregator -->|"uses blocks"| node_transformer
node_model -.->|"predicts cameras"| node_camera_head
node_model -.->|"predicts depth"| node_dense_head
node_model -.->|"aligns text"| node_text_head
node_camera_head -.->|"encodes pose"| node_pose_decode
node_pose_decode -->|"converts rotations"| node_rotation
node_user -->|"uploads media"| node_demo_app
node_demo_app -->|"loads images"| node_image_prep
node_demo_app -->|"runs inference"| node_model
node_demo_app -->|"decodes cameras"| node_pose_decode
node_demo_app -->|"renders scene"| node_visualization
node_trainer -->|"instantiates dataset"| node_datasets
node_datasets -->|"uses base dataset"| node_dataset_base
node_trainer -.->|"computes objective"| node_losses
node_trainer -->|"saves state"| node_checkpoint
node_curation -.->|"reads sequences"| node_datasets
node_eval -.->|"computes scores"| node_eval_metrics

click node_image_prep "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/load_fn.py"
click node_model "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/vggt_omega.py"
click node_aggregator "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/aggregator.py"
click node_transformer "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/layers/vision_transformer.py"
click node_camera_head "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/camera_head.py"
click node_dense_head "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/dense_head.py"
click node_text_head "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/text_alignment_head.py"
click node_pose_decode "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/pose_enc.py"
click node_rotation "https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/rotation.py"
click node_demo_app "https://github.com/facebookresearch/vggt-omega/blob/main/demo_gradio.py"
click node_visualization "https://github.com/facebookresearch/vggt-omega/blob/main/visual_util.py"
click node_trainer "https://github.com/facebookresearch/vggt-omega/blob/main/training/trainer.py"
click node_datasets "https://github.com/facebookresearch/vggt-omega/blob/main/training/data/datasets/unified.py"
click node_dataset_base "https://github.com/facebookresearch/vggt-omega/blob/main/training/data/base_dataset.py"
click node_losses "https://github.com/facebookresearch/vggt-omega/blob/main/training/losses/loss.py"
click node_checkpoint "https://github.com/facebookresearch/vggt-omega/blob/main/training/train_utils/checkpoint.py"
click node_curation "https://github.com/facebookresearch/vggt-omega/blob/main/training/curation/compute_metrics.py"
click node_eval "https://github.com/facebookresearch/vggt-omega/blob/main/eval/evaluate.py"
click node_eval_metrics "https://github.com/facebookresearch/vggt-omega/blob/main/eval/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_image_prep,node_model,node_aggregator,node_transformer,node_user toneBlue
class node_camera_head,node_dense_head,node_text_head,node_pose_decode,node_rotation toneAmber
class node_demo_app,node_visualization toneMint
class node_trainer,node_datasets,node_dataset_base,node_losses,node_checkpoint toneRose
class node_curation,node_eval,node_eval_metrics toneIndigo
```

## Components

### Inference Model

- [Image Preprocessing](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/load_fn.py): `vggt_omega/utils/load_fn.py`
- [VGGT-Ω Model](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/vggt_omega.py): `vggt_omega/models/vggt_omega.py`
- [Frame Aggregator](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/aggregator.py): `vggt_omega/models/aggregator.py`
- [Vision Transformer](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/layers/vision_transformer.py): `vggt_omega/models/layers/vision_transformer.py`

### Geometry Outputs

- [Camera Head](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/camera_head.py): `vggt_omega/models/heads/camera_head.py`
- [Dense Prediction Head](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/dense_head.py): `vggt_omega/models/heads/dense_head.py`
- [Text Alignment Head](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/models/heads/text_alignment_head.py): `vggt_omega/models/heads/text_alignment_head.py`
- [Pose Decoder](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/pose_enc.py): `vggt_omega/utils/pose_enc.py`
- [Rotation Utilities](https://github.com/facebookresearch/vggt-omega/blob/main/vggt_omega/utils/rotation.py): `vggt_omega/utils/rotation.py`

### Interactive Demo

- [Gradio Demo](https://github.com/facebookresearch/vggt-omega/blob/main/demo_gradio.py): `demo_gradio.py`
- [3D Visualization](https://github.com/facebookresearch/vggt-omega/blob/main/visual_util.py): `visual_util.py`

### Model Training

- [Training Orchestrator](https://github.com/facebookresearch/vggt-omega/blob/main/training/trainer.py): `training/trainer.py`
- [Unified Datasets](https://github.com/facebookresearch/vggt-omega/blob/main/training/data/datasets/unified.py): `training/data/datasets/unified.py`
- [Dataset Utilities](https://github.com/facebookresearch/vggt-omega/blob/main/training/data/base_dataset.py): `training/data/base_dataset.py`
- [Training Losses](https://github.com/facebookresearch/vggt-omega/blob/main/training/losses/loss.py): `training/losses/loss.py`
- [Checkpoint Handling](https://github.com/facebookresearch/vggt-omega/blob/main/training/train_utils/checkpoint.py): `training/train_utils/checkpoint.py`

### Data Curation & Evaluation

- [Geometric Curation Metrics](https://github.com/facebookresearch/vggt-omega/blob/main/training/curation/compute_metrics.py): `training/curation/compute_metrics.py`
- [Benchmark Evaluation](https://github.com/facebookresearch/vggt-omega/blob/main/eval/evaluate.py): `eval/evaluate.py`
- [Evaluation Metrics](https://github.com/facebookresearch/vggt-omega/blob/main/eval/metrics.py): `eval/metrics.py`

### Other components

- **Image / Video User**

## Connections

- Image / Video User → Image Preprocessing: provides images (evidence: `README.md`)
- Image Preprocessing → VGGT-Ω Model: prepares tensors (evidence: `README.md`)
- VGGT-Ω Model → Frame Aggregator: encodes frames
- Frame Aggregator → Vision Transformer: uses blocks (evidence: `vggt_omega/models/aggregator.py`)
- VGGT-Ω Model → Camera Head: predicts cameras
- VGGT-Ω Model → Dense Prediction Head: predicts depth
- VGGT-Ω Model → Text Alignment Head: aligns text
- Camera Head → Pose Decoder: encodes pose
- Pose Decoder → Rotation Utilities: converts rotations (evidence: `vggt_omega/utils/pose_enc.py`)
- Image / Video User → Gradio Demo: uploads media (evidence: `README.md`)
- Gradio Demo → Image Preprocessing: loads images (evidence: `demo_gradio.py`)
- Gradio Demo → VGGT-Ω Model: runs inference (evidence: `README.md`)
- Gradio Demo → Pose Decoder: decodes cameras (evidence: `demo_gradio.py`)
- Gradio Demo → 3D Visualization: renders scene (evidence: `demo_gradio.py`)
- Training Orchestrator → Unified Datasets: instantiates dataset (evidence: `training/trainer.py`)
- Unified Datasets → Dataset Utilities: uses base dataset (evidence: `training/data/datasets/unified.py`)
- Training Orchestrator → Training Losses: computes objective
- Training Orchestrator → Checkpoint Handling: saves state (evidence: `training/trainer.py`)
- Geometric Curation Metrics → Unified Datasets: reads sequences
- Benchmark Evaluation → Evaluation Metrics: computes scores
