# fkryan/gazelle architecture

> Architecture diagram of the GitHub repository fkryan/gazelle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fkryan/gazelle
- Repository: https://github.com/fkryan/gazelle

## Overview

##### Main Components

a. Input Processing Layer:
- Image Input (448x448 RGB)
- Input Transform Module

b. Core Architecture Components:
- Frozen DINOv2 Backbone (ViT-B/ViT-L variants)
- Lightweight Gaze Decoder
- Multi-head Attention Mechanism

c. Output Components:
- Spatial Heatmap (64x64)
- In/Out Frame Prediction (for specific models)

**[Input Layer]**

**[Backbone Layer]**

**[Decoder Layer]**

**[Output Layer]**

##### Specific Elements

a. Input Section:
- Transformation pipeline

b. Backbone Section:
- ViT architecture structure
- Feature extraction points

c. Decoder Section:
- Lightweight decoder architecture
- Attention mechanisms
- Learned parameters vs. frozen ones

d. Output Section:
- Heatmap generation
- Optional in/out prediction branch

##### Important Details to Highlight

- Parameter counts to emphasize the lightweight nature
- Feature dimensions at key points
- Attention flow
- Where xformers optimization can be applied

##### Technical Specifications

- Input image size (448x448)
- Output heatmap size (64x64)
- Backbone variants (ViT-B/ViT-L)

##### Implementation Notes

Callouts for:
- PyTorch Hub integration points
- xformers optimization locations
- Multi-person inference capability
- Batch processing support

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input
        IMG[Image Input 448x448]:::input
        BB[Bounding Box Input]:::input
        TRANS[Input Transform]:::input
    end

    subgraph Backbone
        DINO[DINOv2 Backbone<br>ViT-B/ViT-L]:::frozen
    end

    subgraph Decoder
        ATT[Multi-head<br>Attention]:::decoder
        LIGHT[Lightweight<br>Decoder]:::decoder
    end

    subgraph Output
        HEAT[Spatial Heatmap<br>64x64]:::output
        INOUT[In/Out Frame<br>Prediction]:::output
    end

    IMG --> TRANS
    BB --> TRANS
    TRANS --> DINO
    DINO --> ATT
    ATT --> LIGHT
    LIGHT --> HEAT
    LIGHT -.-> INOUT

    %% Component Mappings
    click DINO "https://github.com/fkryan/gazelle/blob/main/gazelle/backbone.py"
    click LIGHT "https://github.com/fkryan/gazelle/blob/main/gazelle/model.py"
    click TRANS "https://github.com/fkryan/gazelle/blob/main/gazelle/utils.py"
    click IMG "https://github.com/fkryan/gazelle/blob/main/data_prep/preprocess_gazefollow.py"
    click BB "https://github.com/fkryan/gazelle/blob/main/data_prep/preprocess_vat.py"
    click HEAT "https://github.com/fkryan/gazelle/blob/main/scripts/eval_gazefollow.py"
    click INOUT "https://github.com/fkryan/gazelle/blob/main/scripts/eval_vat.py"

    %% Legend
    classDef input fill:#a8d1ff,stroke:#333
    classDef frozen fill:#d3d3d3,stroke:#333,stroke-dasharray: 5 5
    classDef decoder fill:#90EE90,stroke:#333
    classDef output fill:#ffb366,stroke:#333

    %% Technical Specifications
    subgraph Legend
        L1[Input Components]:::input
        L2[Frozen Components]:::frozen
        L3[Trainable Components]:::decoder
        L4[Output Components]:::output
    end
```
