# facebookresearch/vggt architecture

> Architecture diagram of the GitHub repository facebookresearch/vggt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/vggt
- Repository: https://github.com/facebookresearch/vggt

## Overview

##### Project Type and Purpose
- This is an open-source machine-learning library and demo tool for 3D scene reconstruction and camera pose estimation. VGGT (“Visual Geometry Grounded Transformer”) is implemented as a Python package that defines a transformer-based model, utility modules, and demo scripts.

##### File Structure Analysis
- Top-level files:
- pyproject.toml, requirements.txt – package and dependency definitions
- README.md, docs/, examples/ – documentation and sample data
- demo_*.py scripts – runnable demos (Gradio, Viser, COLMAP integration)
- visual_util.py – general visualization utilities
- Core package “vggt/” with subpackages:
- dependency/ – non-learning modules for geometric projection, tracking
- layers/ – backbone transformer layers (attention, patch embedding, etc.)
- heads/ – output heads for camera, depth, point, track predictions
- models/ – aggregator and VGGT model orchestration
- utils/ – helpers for loading, pose encoding, geometry, visualization

##### README Insights
- Core pipeline:
1. Load and preprocess images
2. Pass images to `VGGT.aggregator` → fused tokens
3. Feed tokens into various heads (camera, depth, point, track)
4. Postprocess outputs (e.g., unprojection to point clouds)
- Demos integrate Gradio, Viser for interactive visualization
- External dependencies: PyTorch, torchvision, Hugging Face Hub, FlashAttention

a. Main Components
- Input Module – image loader & preprocessor (`vggt.utils.load_fn`)
- Transformer Backbone – patch embedding & multi-view aggregator (`vggt.layers`, `models/aggregator`)
- Output Heads –
­– Camera Head (`heads/camera_head.py`)
­– Depth Head (`heads/depth_head.py`)
­– Point Head (`heads/point_head.py`)
­– Track Head (`heads/track_head.py` + `dependency/track_modules`)
- Post-Processing – pose decoding (`utils/pose_enc`), unprojection (`utils/geometry`), visualization (`utils/visual_track`)
- Demo Interfaces – Gradio (`demo_gradio.py`), Viser (`demo_viser.py`), COLMAP bridge (`demo_colmap.py`)
- External Services – Hugging Face Hub for model weights, third-party rendering libs (Viser, Gradio)

b. Relationships and Data Flow
1. Raw images → Input Module → Preprocessed tensor
2. Tensor → Aggregator → Aggregated tokens
3. Tokens → each Head → Raw outputs (pose encodings, depth maps, point maps, track predictions)
4. Raw outputs → Post-Processing → Usable formats (extrinsic/intrinsic matrices, 3D point clouds, visual tracks)
5. Results → Demo Interfaces → User visualization

c. Architectural Patterns
- Modular design: clear separation between backbone, heads, and utils
- Extensibility:
- `track_modules` submodules implement various tracking refinements
- Utility functions can be replaced or extended for custom processing
- Feed-forward pipeline: single-pass inference

d. Key Technologies
- PyTorch & FlashAttention for transformer efficiency
- Hugging Face Hub for model weight distribution
- Gradio & Viser for interactive demos
- NumPy/OpenCV conventions in geometry utilities

- Rectangles (light blue) for ML modules (Input → Backbone → Heads)
- Rectangles (light green) for Utils/Post-processing
- Rounded rectangles for demo apps
- Images → Preprocessor → Aggregator → Heads → Post-Processing → Visualization

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Inference Pipeline
    Input["Input Module\nBatch×3×H×W"]:::ml
    click Input "https://github.com/facebookresearch/vggt/blob/main/vggt/utils/load_fn.py"

    Layers["Transformer Layers\nPatch Embedding\nMulti-Head Attention"]:::ml
    click Layers "https://github.com/facebookresearch/vggt/tree/main/vggt/layers/"

    Aggregator["VGGT Aggregator\nTokens B×N×D"]:::ml
    click Aggregator "https://github.com/facebookresearch/vggt/blob/main/vggt/models/aggregator.py"

    VGGTModel["VGGT Model Orchestration"]:::ml
    click VGGTModel "https://github.com/facebookresearch/vggt/blob/main/vggt/models/vggt.py"

    %% Heads
    subgraph "Output Heads"
        direction TB
        CameraHead["Camera Head\nPose Encoding"]:::ml
        click CameraHead "https://github.com/facebookresearch/vggt/blob/main/vggt/heads/camera_head.py"

        DepthHead["Depth Head\nDepth Map H×W"]:::ml
        click DepthHead "https://github.com/facebookresearch/vggt/blob/main/vggt/heads/dpt_head.py"

        PointHead["Point Head\nPoints H×W"]:::ml

        TrackHead["Track Head"]:::ml
        click TrackHead "https://github.com/facebookresearch/vggt/blob/main/vggt/heads/track_head.py"
        subgraph "Track Head Details"
            direction TB
            TH_In["Input Tokens"] 
            BasePred["base_track_predictor"] 
            Blocks["Transformer Blocks"] 
            Modules["Dependency Modules"] 
            click Modules "https://github.com/facebookresearch/vggt/tree/main/vggt/dependency/track_modules/"
            TH_Out["Track Predictions"]
            TH_In --> BasePred --> Blocks --> Modules --> TH_Out
        end
    end

    %% Post-Processing
    subgraph "Post-Processing" 
        direction TB
        PoseDec["Pose Decoding\nExtrinsics/Intrinsics"]:::util
        click PoseDec "https://github.com/facebookresearch/vggt/blob/main/vggt/utils/pose_enc.py"

        Geometry["Unprojection & Geometry\n3D Point Clouds"]:::util
        click Geometry "https://github.com/facebookresearch/vggt/blob/main/vggt/utils/geometry.py"

        VisualTrack["Visualize Tracks"]:::util
        click VisualTrack "https://github.com/facebookresearch/vggt/blob/main/vggt/utils/visual_track.py"
    end

    %% Demos
    subgraph "Demo Interfaces"
        direction TB
        Gradio["Gradio App"]:::demo
        click Gradio "https://github.com/facebookresearch/vggt/blob/main/demo_gradio.py"

        Viser["Viser App"]:::demo
        click Viser "https://github.com/facebookresearch/vggt/blob/main/demo_viser.py"

        COLMAP["COLMAP Bridge"]:::demo
        click COLMAP "https://github.com/facebookresearch/vggt/blob/main/demo_colmap.py"
    end

    %% External Services & Resources
    Hub(("Hugging Face Hub")):::external
    Examples["examples/"]:::resource
    Docs["docs/"]:::resource

    %% Build & Deployment
    subgraph "Build & Deployment"
        direction TB
        Req["requirements.txt"]:::build
        PyProj["pyproject.toml"]:::build
    end

    %% Connections
    Input -->|"Preprocessed Tensor"| Layers -->|"Feature Tokens"| Aggregator -->|"Aggregated Tokens"| VGGTModel
    VGGTModel --> CameraHead -->|"Raw Pose Encodings"| PoseDec
    VGGTModel --> DepthHead -->|"Raw Depth Map"| Geometry
    VGGTModel --> PointHead -->|"Raw Points"| Geometry
    VGGTModel --> TrackHead -->|"Raw Track Tokens"| TH_In

    PoseDec --> Gradio
    Geometry --> Gradio
    VisualTrack --> Gradio

    PoseDec --> Viser
    Geometry --> Viser
    VisualTrack --> Viser

    PoseDec --> COLMAP
    Geometry --> COLMAP
    VisualTrack --> COLMAP

    %% External
    Hub -.-> VGGTModel
    Examples -.-> Input
    Docs -.-> Input

    %% Styles
    classDef ml fill:#ADD8E6,stroke:#333,stroke-width:1px;
    classDef util fill:#90EE90,stroke:#333,stroke-width:1px;
    classDef demo fill:#FFD700,stroke:#333,stroke-width:1px;
    classDef external fill:#FFE4E1,stroke:#333,stroke-width:1px;
    classDef resource fill:#D3D3D3,stroke:#333,stroke-width:1px;
    classDef build fill:#E6E6FA,stroke:#333,stroke-width:1px;
```
