How microsoft/videox works
1. This repository is a monorepo of four mostly independent PyTorch research projects under the “VideoX” umbrella: 2D-TAN, MS-2D-TAN, SeqTrack, and X-CLIP. The top-level README is the only real cross-project entry point; each subdirectory has its own training, evaluation, configuration, and data layout.
2. The shared architectural pattern across all four projects is:
- dataset-specific preprocessing and split files
- YAML experiment/config files
- a model library with modular subcomponents
- training and testing entrypoints
- benchmark-specific evaluation code
This makes each paper implementation self-contained rather than sharing a single runtime.
3. 2D-TAN is the earliest moment-localization implementation. Its core pipeline is: text query + precomputed video features → temporal proposal map construction → 2D temporal adjacency modeling → scoring candidate moments. The code is organized around lib/core for config/engine/eval utilities, lib/datasets for ActivityNet/Charades-STA/TACoS loaders, and lib/models for frame, fusion, proposal, map, and loss modules. Training and test are launched from moment_localization/train.py and moment_localization/test.py.
4. 2D-TAN’s model boundary is modular: frame-level pooling lives in frame_modules, language/video fusion in fusion_modules, temporal proposal generation in prop_modules, and 2D convolution over the moment map in map_modules. The final network is assembled in lib/models/tan.py, which is the main model composition point.
5. MS-2D-TAN extends the 2D-TAN architecture to multiple temporal scales. Compared with 2D-TAN, it adds scale-aware proposal reasoning and extra prediction/fusion structure. The main additions are clip_modules, pred_modules, multiscale proposal logic, and video_text_fusion, while keeping the same overall data flow: query/video features in, multi-scale temporal map reasoning, localized moment outputs.
6. MS-2D-TAN keeps the same dataset family and evaluation problem as 2D-TAN, but its experiments are more varied through YAMLs for ActivityNet, Charades, and TACoS. It also includes utility scripts for visualization and dataset splitting, indicating a stronger emphasis on inspecting temporal heatmaps and benchmark-specific preprocessing.
7. SeqTrack is architecturally different: it is a visual tracking system, not a moment-localization system. Its core idea is sequence generation of bounding boxes using an encoder-decoder transformer. The primary model lives in lib/models/seqtrack, split into encoder.py, decoder.py, seqtrack.py, and vit.py, with utility code in lib/utils for boxes, positional embeddings, EMA/lr schedules, and tensor helpers.
8. SeqTrack has two parallel application surfaces:
- tracking/ for running and analyzing the SeqTrack tracker
- lib/train and lib/test for the research training/evaluation framework
This separation matters: training uses its own actor/trainer/data abstractions, while testing has benchmark-specific dataset wrappers and tracker entrypoints. The repo also includes external/AR, a large embedded dependency/fork of the pytracking/LTR ecosystem for evaluation and legacy tracking baselines.
9. SeqTrack’s data architecture is broader than the moment-localization projects. It supports image and video tracking datasets such as GOT-10k, LaSOT, TrackingNet, COCO, ImageNetVID, and VOT variants, with both native and LMDB-backed dataset variants. The presence of data_specs, LMDB writers, and transform utilities shows that data storage and sampling are first-class concerns, not just model code.
10. X-CLIP is a standalone video classification system built around CLIP-style vision-language pretraining. Its main flow is: video frames → image/video encoder and temporal cross-frame attention → text prompt generation for class names → fused video-text prediction. The core model code is in models/xclip.py, supported by models/cct.py, models/mit.py, and models/prompt.py, while the pretrained CLIP code and tokenizer are vendored under clip/.
11. X-CLIP’s data layer is classification-oriented rather than retrieval-oriented. It uses datasets/build.py, pipeline.py, blending.py, and rand_augment.py for video augmentation and dataset composition, with label CSVs for HMDB51, Kinetics-400/600, and UCF101. This is a clear boundary from the other projects: inputs are labeled video clips, not paired text queries or tracking sequences.
12. Configuration and experiment management are file-driven in all projects. The experiments/ folders contain one YAML per paper setting or backbone variant, so architecture, dataset choice, and scale parameters are encoded outside the model code. This makes the repo diagram friendly: config files feed into training scripts, which instantiate dataset loaders and model modules.
13. External data and assets are important architectural dependencies. 2D-TAN/MS-2D-TAN rely on precomputed visual features and dataset JSON/CSV annotations; X-CLIP relies on CLIP weights and video-class labels; SeqTrack depends heavily on benchmark datasets and the external tracking framework. The repo is therefore more of an experimental system with dataset preparation steps than a single deployable application.
14. If you draw this repository as an architecture diagram, the main boundary should be between the four project subtrees. Inside each subtree, the dominant flow is: dataset/annotation preparation → configurable training/evaluation entrypoint → modular model components → benchmark-specific outputs and analysis. The largest shared theme is not code reuse, but a common research workflow for video understanding.
Main entry points: tan.py TAN model · README.md Root README · train.py Train · run.py Run
Components 34
VideoX monorepo
Top-level umbrella with four mostly independent research projects.
- Root README · Umbrella documentation and cross-project starting point.
README.md
2D-TAN
Moment localization project for text-conditioned temporal proposal reasoning.
- Train · Launches 2D-TAN training.
2D-TAN/moment_localization/train.py - Test · Runs 2D-TAN evaluation.
2D-TAN/moment_localization/test.py - Core utils · Shared config, engine, evaluation, and helper utilities.
2D-TAN/lib/core - Datasets · ActivityNet, Charades-STA, and TACoS dataset adapters.
2D-TAN/lib/datasets - TAN model · Main network composition for moment localization.
2D-TAN/lib/models/tan.py - Frame pool · Frame-level pooling and feature preparation.
2D-TAN/lib/models/frame_modules/frame_pool.py - Fusion · Combines language and visual features.
2D-TAN/lib/models/fusion_modules/base_fusion.py - Proposal gen · Dense and sparse temporal proposal generation.
2D-TAN/lib/models/prop_modules - Map conv · 2D convolution over the moment map for candidate scoring.
2D-TAN/lib/models/map_modules/map_conv.py - Loss · Objective used for moment localization training.
2D-TAN/lib/models/loss.py
MS-2D-TAN
Multi-scale extension of 2D-TAN for temporal localization.
- Run · Launches MS-2D-TAN experiments and evaluation.
MS-2D-TAN/moment_localization/run.py - Eval · Standalone evaluation workflow for multi-scale experiments.
MS-2D-TAN/moment_localization/eval.py - Core utils · Config, training engine, evaluation, and utilities.
MS-2D-TAN/lib/core - Datasets · Dataset adapters plus transforms for ActivityNet, Charades, and TACoS.
MS-2D-TAN/lib/datasets - MS-TAN · Multi-scale TAN composition point.
MS-2D-TAN/lib/models/TAN.py - Multi-scale · Scale-aware proposal construction and dense/sparse reasoning.
MS-2D-TAN/lib/models/prop_modules/multiscale.py - Video-text fusion · Enhanced query/video fusion for scale-aware localization.
MS-2D-TAN/lib/models/fusion_modules/video_text_fusion.py - Predictors · Additional prediction and aggregation modules for multi-scale outputs.
MS-2D-TAN/lib/models/pred_modules/modules.py - Tools · Heatmap visualization and testset splitting helpers.
MS-2D-TAN/tools
SeqTrack
Transformer-based visual tracking system with training and benchmark evaluation.
- Train · Research training framework for SeqTrack.
SeqTrack/lib/train/train_script.py - Test · Benchmark evaluation and tracker testing framework.
SeqTrack/lib/test/evaluation - Tracking · Tracker execution, profiling, and analysis surface.
SeqTrack/tracking - SeqTrack model · Main sequence-to-box model implementation.
SeqTrack/lib/models/seqtrack/seqtrack.py - Encoder/decoder · Encoder, decoder, and ViT backbone components.
SeqTrack/lib/models/seqtrack - Tracking data · Training datasets, LMDB variants, sampling, and transforms.
SeqTrack/lib/train/dataset - Data specs · Dataset split and class-map specifications.
SeqTrack/lib/train/data_specs - AR fork · Vendored pytracking/LTR ecosystem for legacy training and evaluation.
SeqTrack/external/AR
X-CLIP
CLIP-style video classification system for action recognition.
- Main · Top-level training/evaluation script.
X-CLIP/main.py - X-CLIP model · CLIP-style video classification and prompt-fused prediction.
X-CLIP/models/xclip.py - Vision backbones · Supporting video/image encoder components and prompts.
X-CLIP/models - CLIP fork · Pretrained CLIP implementation and tokenizer.
X-CLIP/clip - Data pipeline · Dataset building, blending, and video augmentation pipeline.
X-CLIP/datasets - Label CSVs · Dataset label mappings for HMDB51, Kinetics, and UCF101.
X-CLIP/labels
Connections 33
- Root README → Train: documents · Root docs point to the project entrypoints.
- Root README → Run: documents · Root docs point to the project entrypoints.
- Root README → Train: documents · Root docs point to the project entrypoints.
- Root README → Main: documents · Root docs point to the project entrypoints.
- Train → Core utils: uses · Training relies on shared config, engine, and evaluation helpers.
- Test → Core utils: uses · Testing relies on shared config, engine, and evaluation helpers.
- Core utils → Datasets: loads · Core runtime instantiates dataset adapters.
- Train → TAN model: builds · Training constructs the TAN network.
- TAN model → Frame pool: contains · Frame-level pooling is part of the model composition.
- TAN model → Fusion: contains · Language-video fusion is part of the model composition.
- TAN model → Proposal gen: contains · Temporal proposal generation is part of the model composition.
- TAN model → Map conv: contains · 2D moment map reasoning is part of the model composition.
- Train → Loss: optimizes · Training uses the localization loss.
- Run → Core utils: uses · Run script relies on shared config, engine, and evaluation helpers.
- Eval → Core utils: uses · Evaluation script relies on shared config, engine, and evaluation helpers.
- Core utils → Datasets: loads · Core runtime instantiates dataset adapters and transforms.
- Run → MS-TAN: builds · Experiments construct the multi-scale TAN network.
- MS-TAN → Multi-scale: contains · Multi-scale proposal reasoning is a central architectural addition.
- MS-TAN → Video-text fusion: contains · Video-text fusion is a central architectural addition.
- MS-TAN → Predictors: contains · Prediction heads refine localized outputs.
- Run → Tools: analyzes · Visualization and split scripts support experiment inspection.
- Train → Tracking data: samples · Training uses tracking dataset loaders and augmentation pipelines.
- Train → SeqTrack model: optimizes · Training uses the SeqTrack model.
- SeqTrack model → Encoder/decoder: contains · Encoder-decoder transformer and ViT backbone make up the model core.
- Tracking data → Data specs: driven by · Training data loading is guided by dataset split specs.
- Test → AR fork: integrates · Benchmark evaluation depends on the vendored AR/pytracking ecosystem.
- Tracking → SeqTrack model: runs · Tracker surface uses the SeqTrack model for analysis and inference.
- Tracking → AR fork: extends · Tracker tooling interoperates with the external AR stack.
- Main → Data pipeline: builds · Main script constructs classification datasets and pipelines.
- Main → X-CLIP model: trains · Main script trains or evaluates the X-CLIP model.
- X-CLIP model → Vision backbones: uses · Model combines video encoders and prompt modules.
- X-CLIP model → CLIP fork: relies on · Model depends on vendored CLIP components.
- Data pipeline → Label CSVs: aligns with · Pipeline is driven by dataset label CSVs.