How navi-challenge/navi-2025 works
Project Type and Purpose
• This is a deep‐learning research codebase for Video‐Text Retrieval (VTR) on the NAVI dataset (text‐based vehicle retrieval).
• It implements data preprocessing, PyTorch dataloaders, model definitions (XCLIP, CLIP, cross‐modal transformers), training and evaluation scripts, and submission‐generation utilities.
File Structure Analysis
Top‐level components:
• data_navi/ – raw NAVI dataset with subfolders for data, input, output, logs, models, similarity matrices, submissions
• dataloaders/ – PyTorch dataloader implementations for various benchmarks including NAVI
• modules/ – core model code: tokenizers, CLIP/XCLIP modules, cross‐modal modules, optimization, file utilities, configs
• main_clip4clip.py & main_xclip.py – entrypoints for training/testing corresponding model variants
• preprocess_data.py – transforms raw JSON/video data into the required format
• make_submission_from_sim_matrix.py – post‐processing script for submission files
• metrics.py & util.py – evaluation metrics and helper functions
• scripts/ – shell scripts to launch training/testing
• requirements.txt – Python dependencies
Key Insights from README
• Preprocess → train/test with shell scripts
• External pretrained weights for XCLIP, ViT‐B/16
• PyTorch 1.7.1 core framework
Main Components
A. Data Layer
1. Raw Dataset (data_navi/data & data_navi/navi/input/json)
2. Preprocessing Module (preprocess_data.py) → produces structured data for training/testing
B. Data Access Layer
1. dataloaders package
2. Video utilities (rawvideo_util.py)
3. JSON parsers for tracks & queries
C. Model Layer
1. Tokenization (modules/tokenization_clip.py)
2. Vision encoder (pretrained ViT‐B/16) – modules/module_clip.py
3. Text encoder (CLIP extended to max length 256) – modules/module_clip.py
4. Cross‐modal fusion (modules/module_cross.py & cross-base)
5. Configuration (modules/cross_config.json, until_config.py)
D. Training & Optimization
1. Training scripts (main_clip4clip.py, main_xclip.py)
2. Optimization module (modules/optimization.py)
3. Hyperparameter configs via shell scripts (scripts/*.sh)
4. External weights loader (modules/file_utils.py)
E. Evaluation & Submission
1. Similarity matrix generator → data_navi/navi/output/sim_matrix
2. make_submission_from_sim_matrix.py → data_navi/navi/output/submissions
3. metrics.py calculates retrieval metrics → data_navi/navi/output/log
F. Utilities & Support
1. util.py – general-purpose helpers
2. requirements.txt – environment specification
Relationships & Data Flow
• Raw Dataset → Preprocessing → Structured JSON/video frames
• Structured data → Dataloaders → Batched inputs
• Batched inputs → Model encoders (vision & text) → Feature embeddings
• Embeddings → Cross‐modal module → Similarity scores
• Scores → Loss & optimizer → Model weights checkpointing (data_navi/navi/output/models)
• At test time: Scores → sim_matrix → make_submission → final submission file
• metrics.py invoked within training/test loops to log performance
Architectural Patterns & Principles
• Modular separation: data ingestion, processing, modeling, training, evaluation
• Pipeline pattern: clear stages (preprocess → load → train → evaluate → submit)
• Config-driven: JSON configs and shell scripts parameterize experiments
• Reusable utilities: central modules for tokenization, file I/O, optimization