How mhnazeri/vanp works
Project Type and Purpose
• The project is a machine learning research codebase focused on self-supervised vision-action pre-training for navigation tasks.
• Its purpose is to train a model (VANP) that learns which visual cues are most useful for navigation. It does this using a self-supervised pretext task and then applies the pre-trained model for downstream navigation tasks.
• Key functionalities include parsing navigation datasets (MuSoHu and SCAND), applying data augmentation, and training two types of tasks: (a) a pretext training phase and (b) a downstream navigation training/validation phase.
Analyzing the File Structure
• Top-Level Files: LICENSE.txt, README.md, requirements.txt, and run.sh (which acts as an entry point for running training commands).
• The VANP Directory acts as the core directory, organized into several sub-components:
• conf/:
• Contains YAML configuration files (config_pretext.yaml, parser.yaml) that manage hyperparameters and dataset parsing settings.
• model/:
• Houses the model definitions including core models (model.py), an autoencoder (autoencoder.py), a policy network for navigation (policynet.py), a pretext model (pretext_model.py), and dataloader functions.
• utils/:
• Includes a range of utility scripts for data augmentation (augmenter.py), augmenting training using techniques like LARS and custom loss functions, parsing specific dataset formats (musohu_parser.py, parser.py, scand_parser.py), and additional helper functions.
• The VANP directory also contains the main training script (train_pretext.py) that has code related to both pretext training and the downstream navigation task.
• The docs folder contains documentation, sample data parser instructions, and sample outputs. This reinforces that dataset parsing and model evaluation are core components.
Key Components
a. Configuration Management:
• Configuration files (config_pretext.yaml, parser.yaml) as a component that feeds into the rest of the system. Indicate that OmegaConf is used to load these configurations.
b. Data Input & Parsing:
c. Model Architecture and Components:
• Pretext Model: Represent the pretext encoder(s) (likely using Transformer Encoder blocks) that learn embeddings from a history of visual observations, future actions, and a goal image.
• Action/Policy Network: Indicate the role of the policynet module to connect the learned embeddings to navigation decisions.
• Auxiliary Modules: Include the autoencoder (for possible reconstruction tasks) and any neural network blocks defined in nn.py.
• Mutual Information Maximization: While not a separate file, note that this objective is central to model training, so an annotation should highlight that this objective function (implemented via loss functions in utils/loss.py) tightly couples the model components.
• Use clear labels (e.g., “Vision-Action Pretext Module”, “Downstream Navigation Module”) for clarity.
d. Training Pipelines:
• Two training pipelines:
• Pretext Training Pipeline: Use components defined in train_pretext.py, which include data loader, model forward passes, loss computation (including mutual information maximization), and gradient updates.
• Downstream Navigation Training/Validation Pipeline: Represent this pipeline that utilizes the pre-trained model and integrates the policy network for navigation decisions.
e. External Dependencies and Monitoring:
• Integration with Comet.ml for logging and experiment tracking.
• Additional frameworks/libraries such as OmegaConf for configuration management.
Detailed Elements
• Each component should have a clear name and role.
• Design choices like the use of Transformer Encoders for embedding, mutual information maximization, and augmentation strategies that enhance the training process.