# mhnazeri/vanp architecture

> Architecture diagram of the GitHub repository mhnazeri/vanp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mhnazeri/vanp
- Repository: https://github.com/mhnazeri/vanp

## Overview

##### Project Type and Purpose
- The project is a machine learning research codebase focused on self-supervised vision-action pre-training for navigation tasks.
- Its purpose is to train a model (VANP) that learns which visual cues are most useful for navigation. It does this using a self-supervised pretext task and then applies the pre-trained model for downstream navigation tasks.
- Key functionalities include parsing navigation datasets (MuSoHu and SCAND), applying data augmentation, and training two types of tasks: (a) a pretext training phase and (b) a downstream navigation training/validation phase.

##### Analyzing the File Structure
- Top-Level Files: LICENSE.txt, README.md, requirements.txt, and run.sh (which acts as an entry point for running training commands).
- The VANP Directory acts as the core directory, organized into several sub-components:
- conf/:
- Contains YAML configuration files (config_pretext.yaml, parser.yaml) that manage hyperparameters and dataset parsing settings.
- model/:
- Houses the model definitions including core models (model.py), an autoencoder (autoencoder.py), a policy network for navigation (policynet.py), a pretext model (pretext_model.py), and dataloader functions.
- utils/:
- Includes a range of utility scripts for data augmentation (augmenter.py), augmenting training using techniques like LARS and custom loss functions, parsing specific dataset formats (musohu_parser.py, parser.py, scand_parser.py), and additional helper functions.
- The VANP directory also contains the main training script (train_pretext.py) that has code related to both pretext training and the downstream navigation task.
- The docs folder contains documentation, sample data parser instructions, and sample outputs. This reinforces that dataset parsing and model evaluation are core components.

##### Key Components
a. Configuration Management:
- Configuration files (config_pretext.yaml, parser.yaml) as a component that feeds into the rest of the system. Indicate that OmegaConf is used to load these configurations.

b. Data Input & Parsing:

c. Model Architecture and Components:
- Pretext Model: Represent the pretext encoder(s) (likely using Transformer Encoder blocks) that learn embeddings from a history of visual observations, future actions, and a goal image.
- Action/Policy Network: Indicate the role of the policynet module to connect the learned embeddings to navigation decisions.
- Auxiliary Modules: Include the autoencoder (for possible reconstruction tasks) and any neural network blocks defined in nn.py.
- Mutual Information Maximization: While not a separate file, note that this objective is central to model training, so an annotation should highlight that this objective function (implemented via loss functions in utils/loss.py) tightly couples the model components.
- Use clear labels (e.g., “Vision-Action Pretext Module”, “Downstream Navigation Module”) for clarity.

d. Training Pipelines:
- Two training pipelines:
- Pretext Training Pipeline: Use components defined in train_pretext.py, which include data loader, model forward passes, loss computation (including mutual information maximization), and gradient updates.
- Downstream Navigation Training/Validation Pipeline: Represent this pipeline that utilizes the pre-trained model and integrates the policy network for navigation decisions.

e. External Dependencies and Monitoring:
- Integration with Comet.ml for logging and experiment tracking.
- Additional frameworks/libraries such as OmegaConf for configuration management.

##### Detailed Elements
- Each component should have a clear name and role.
- Design choices like the use of Transformer Encoders for embedding, mutual information maximization, and augmentation strategies that enhance the training process.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data and Dependencies
    A["External Datasets (MuSoHu & SCAND)"]:::external
    J["External Dependencies<br/>(PyTorch, OmegaConf, Comet.ml)"]:::external

    %% Configuration Management Subgraph
    subgraph "Configuration Management"
        B1["config_pretext.yaml"]:::config
        B2["parser.yaml"]:::config
    end

    %% Data Parsing & Augmentation Subgraph
    subgraph "Data Parsing & Augmentation"
        C1["musohu_parser.py"]:::dataparse
        C2["scand_parser.py"]:::dataparse
        C3["parser.py"]:::dataparse
        C4["augmenter.py"]:::dataparse
    end

    %% Aggregated Parsed Data Node
    P["Parsed Data"]:::dataparse

    %% Data Loading Node
    D["Data Loader"]:::dataparse

    %% Model Architecture Subgraph
    subgraph "Model Architecture"
        E["Pretext Model<br/>(Transformer Encoders)"]:::model
        F["Policy Network<br/>(Navigation)"]:::model
        subgraph "Auxiliary Modules"
            G1["autoencoder.py"]:::model
            G2["nn.py"]:::model
        end
    end

    %% Training Pipelines and Loss Functions
    H["Training Pipelines<br/>(Pretext & Downstream)"]:::training
    I["Loss Functions<br/>(Mutual Information)"]:::training

    %% Data Flow Connections
    A -->|"rawData"| P
    C1 --> P
    C2 --> P
    C3 --> P
    C4 --> P

    %% Configuration feeding Data Loader and Training
    B1 -->|"configs"| D
    B2 -->|"configs"| D
    B1 -->|"trainConfig"| H
    B2 -->|"trainConfig"| H

    %% Parsed Data to Data Loader
    P -->|"loadedData"| D

    %% Data Loader to Model Components
    D -->|"inputData"| E

    %% Model Internal Flow
    E -->|"embeddings"| F

    %% Training Pipeline connections
    H -->|"update"| E
    H -->|"update"| F
    H -->|"computeLoss"| I
    I -->|"backprop"| E
    I -->|"backprop"| F

    %% External Dependencies influence components
    J --> D
    J --> E
    J --> F
    J --> H

    %% Styles
    classDef config fill:#FFD700,stroke:#DAA520,stroke-width:2px;
    classDef dataparse fill:#66CCFF,stroke:#3399FF,stroke-width:2px;
    classDef model fill:#99FF99,stroke:#33AA33,stroke-width:2px;
    classDef training fill:#FF9999,stroke:#FF4444,stroke-width:2px;
    classDef external fill:#CCCCCC,stroke:#888888,stroke-width:2px;

    %% Click Events
    click B1 "https://github.com/mhnazeri/vanp/blob/main/VANP/conf/config_pretext.yaml"
    click B2 "https://github.com/mhnazeri/vanp/blob/main/VANP/conf/parser.yaml"
    click C1 "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/musohu_parser.py"
    click C2 "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/scand_parser.py"
    click C3 "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/parser.py"
    click C4 "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/augmenter.py"
    click E "https://github.com/mhnazeri/vanp/blob/main/VANP/model/pretext_model.py"
    click F "https://github.com/mhnazeri/vanp/blob/main/VANP/model/policynet.py"
    click G1 "https://github.com/mhnazeri/vanp/blob/main/VANP/model/autoencoder.py"
    click G2 "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/nn.py"
    click D "https://github.com/mhnazeri/vanp/blob/main/VANP/model/dataloader.py"
    click H "https://github.com/mhnazeri/vanp/blob/main/VANP/train_pretext.py"
    click I "https://github.com/mhnazeri/vanp/blob/main/VANP/utils/loss.py"
```
