# google-deepmind/tapnet architecture

> Architecture diagram of the GitHub repository google-deepmind/tapnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google-deepmind/tapnet
- Repository: https://github.com/google-deepmind/tapnet

## Overview

##### Project type and purpose
- This is an open-source Python research library and benchmark suite for “Tracking Any Point” in video (TAP-Vid, TAPIR, RoboTAP, BootsTAPIR, TAPVid-3D, TAPNext, TRAJAN).
- It provides datasets, training and evaluation frameworks, multiple model implementations (JAX + PyTorch), demos (Colab notebooks + live scripts), utilities, and configuration files.

##### Analysis of the file structure
- Top level:
- configs/ – configuration scripts for different models (TAPIR, causal TAPIR, TAPNet, TAPNext)
- tapnet/ – core Python package
– models/ – backbone and tracker model definitions (ResNet, SSM-ViT, TAPIR, TAPNet, TSM, VideoSSM)
– tapir_clustering & robotap/ – clustering code for RoboTAP
– tapnext/ – code, losses, utilities for TAPNext
– tapvid/ – 2D dataset helpers, download, visualization
– tapvid3d/ – 3D annotation generation, splits, evaluation
– training/ – JAX-based training/evaluation pipelines
– torch/ – PyTorch re-implementations of models & utilities
– trajan/ – trajectory autoencoder
– utils/ – common experiment, model, optimizer, transform, and viz utilities
– live_demo.py & pytorch_live_demo.py – real-time inference scripts
- colabs/ – Jupyter notebooks for online demos
- requirements*.txt & pyproject.toml – dependencies
- CI, license, README, CONTRIBUTING, and root-level scripts for cloning, installing, and checkpoint download.

##### Key insights from README
- Multi-model two-stage pipelines (matching → refinement for TAPIR; next-token prediction for TAPNext; autoencoding for TRAJAN).
- Separate benchmarks: TAP-Vid (2D), RoboTAP (robotics), TAPVid-3D.
- Demos run as notebooks or CLI scripts.
- Training in JAX (tapnet/training) and optional PyTorch external training.

##### Detail
- Split “Core Library” into submodules (models, utils, torch, tapnext, trajan, robotap).
- Two stages inside TAPIR block: “Matching Stage” → “Refinement Stage”.
- Optional PyTorch re-implementation path parallel to JAX.
- Notebooks separately, pointing to core library via PYTHONPATH/pip.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        DS1["TAP-Vid Dataset"]:::data
        DS2["RoboTAP Dataset"]:::data
        DS3["Kubric (Synthetic)"]:::data
        DS4["TAPVid-3D Dataset"]:::data
        DS5["External Video (Camera)"]:::data
    end

    %% Configuration Layer
    Conf["Configuration Layer\n(configs/)"]:::config

    %% Core Library
    subgraph "Core Library"
        direction TB
        subgraph "Data Utilities"
            direction TB
            DH["TAP-Vid Dataset Helpers"]:::data
            DH2["TAPVid-3D Annotation Generation"]:::data
        end
        subgraph Models
            direction TB
            M1["TAPNet Models (JAX)"]:::model
            M2["PyTorch Re-implementations"]:::model
            M3["TAPNext Module & Losses"]:::model
            M4["TRAJAN Autoencoder"]:::model
            M5["RoboTAP Clustering Code"]:::model
        end
        U["Core Utilities"]:::util
    end

    %% Training Pipeline
    T1["JAX Training Pipeline"]:::train

    %% Evaluation & Metrics
    subgraph "Evaluation & Metrics"
        direction TB
        E1["TAP-Vid Evaluation Datasets"]:::eval
        E2["TAPVid-3D Evaluation & Metrics"]:::eval
    end

    %% Inference & Visualization
    subgraph "Inference & Visualization"
        direction TB
        I1["Live Demo (JAX)"]:::infer
        I2["Live Demo (PyTorch)"]:::infer
        VI["Visualization Utilities"]:::vis
    end

    %% Demos
    Colab["Colab Demo Notebooks"]:::demo

    %% Checkpoint Storage
    Chk["Checkpoint Storage\n(local/GCS)"]:::storage

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        Ext1["JAX"]:::external
        Ext2["PyTorch"]:::external
        Ext3["Kubric Reader"]:::external
        Ext4["Google Cloud Storage"]:::external
    end

    %% Connections
    DS1 -->|loads data| T1
    DS2 -->|loads data| T1
    DS3 -->|loads data| T1
    DS4 -->|loads data| T1
    DS5 -->|live feed| I1

    T1 -->|"saves checkpoints"| Chk
    Conf -->|configures| T1
    Conf -->|configures| I1
    Conf -->|configures| I2

    DS1 -->|evaluation data| E1
    DS4 -->|evaluation data| E2

    M1 --> T1
    U --> T1

    E1 -->|"computes metrics"| E2
    E2 -->|"reports"| T1

    Chk -->|checkpoints & configs| I1
    Chk -->|checkpoints & configs| I2
    U --> I1
    U --> I2

    I1 --> VI
    I2 --> VI
    VI --> Colab

    Colab --> U
    Colab --> M1

    Ext1 --> T1
    Ext2 --> I2
    Ext3 --> DS3
    Ext4 --> Chk

    %% Click Events
    click Conf "https://github.com/google-deepmind/tapnet/tree/main/configs/"
    click DH "https://github.com/google-deepmind/tapnet/blob/main/tapnet/tapvid/generate_tapvid.py"
    click DH2 "https://github.com/google-deepmind/tapnet/tree/main/tapnet/tapvid3d/annotation_generation/"
    click E1 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/tapvid/evaluation_datasets.py"
    click E2 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/tapvid3d/evaluation/metrics.py"
    click M1 "https://github.com/google-deepmind/tapnet/tree/main/tapnet/models/"
    click M2 "https://github.com/google-deepmind/tapnet/tree/main/tapnet/torch/"
    click M3 "https://github.com/google-deepmind/tapnet/tree/main/tapnet/tapnext/"
    click M4 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/trajan/track_autoencoder.py"
    click M5 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/robotap/tapir_clustering.py"
    click U "https://github.com/google-deepmind/tapnet/tree/main/tapnet/utils/"
    click T1 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/training/experiment.py"
    click T1 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/training/supervised_point_prediction.py"
    click I1 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/live_demo.py"
    click I2 "https://github.com/google-deepmind/tapnet/blob/main/tapnet/pytorch_live_demo.py"
    click Colab "https://github.com/google-deepmind/tapnet/tree/main/colabs/"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085;
    classDef config fill:#e2e3e5,stroke:#6c757d,color:#383d41;
    classDef model fill:#d4edda,stroke:#155724,color:#155724;
    classDef util fill:#f0f0f0,stroke:#adb5bd,color:#495057;
    classDef train fill:#fff3cd,stroke:#856404,color:#856404;
    classDef eval fill:#ffe5b4,stroke:#cc7a00,color:#cc7a00;
    classDef infer fill:#d6d8db,stroke:#1b1e21,color:#1b1e21;
    classDef vis fill:#f5d0a9,stroke:#873e23,color:#873e23;
    classDef demo fill:#e2d4f0,stroke:#5a3e85,color:#5a3e85;
    classDef storage fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24;
```
