How google-deepmind/tapnet works
Project type and purpose
• This is an open-source Python research library and benchmark suite for “Tracking Any Point” in video (TAP-Vid, TAPIR, RoboTAP, BootsTAPIR, TAPVid-3D, TAPNext, TRAJAN).
• It provides datasets, training and evaluation frameworks, multiple model implementations (JAX + PyTorch), demos (Colab notebooks + live scripts), utilities, and configuration files.
Analysis of the file structure
• Top level:
• configs/ – configuration scripts for different models (TAPIR, causal TAPIR, TAPNet, TAPNext)
• tapnet/ – core Python package
– models/ – backbone and tracker model definitions (ResNet, SSM-ViT, TAPIR, TAPNet, TSM, VideoSSM)
– tapir_clustering & robotap/ – clustering code for RoboTAP
– tapnext/ – code, losses, utilities for TAPNext
– tapvid/ – 2D dataset helpers, download, visualization
– tapvid3d/ – 3D annotation generation, splits, evaluation
– training/ – JAX-based training/evaluation pipelines
– torch/ – PyTorch re-implementations of models & utilities
– trajan/ – trajectory autoencoder
– utils/ – common experiment, model, optimizer, transform, and viz utilities
– live_demo.py & pytorch_live_demo.py – real-time inference scripts
• colabs/ – Jupyter notebooks for online demos
• requirements*.txt & pyproject.toml – dependencies
• CI, license, README, CONTRIBUTING, and root-level scripts for cloning, installing, and checkpoint download.
Key insights from README
• Multi-model two-stage pipelines (matching → refinement for TAPIR; next-token prediction for TAPNext; autoencoding for TRAJAN).
• Separate benchmarks: TAP-Vid (2D), RoboTAP (robotics), TAPVid-3D.
• Demos run as notebooks or CLI scripts.
• Training in JAX (tapnet/training) and optional PyTorch external training.
Detail
• Split “Core Library” into submodules (models, utils, torch, tapnext, trajan, robotap).
• Two stages inside TAPIR block: “Matching Stage” → “Refinement Stage”.
• Optional PyTorch re-implementation path parallel to JAX.
• Notebooks separately, pointing to core library via PYTHONPATH/pip.