# marsaki/etpnav architecture

> Architecture diagram of the GitHub repository marsaki/etpnav, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marsaki/etpnav
- Repository: https://github.com/marsaki/etpnav

## Overview

##### Project Overview and Purpose
- The project is a full-stack research system that integrates simulation, perception, planning, and control for vision-language navigation in continuous environments.
- Its aim is to enable an agent to generate long-range navigation plans through online topological mapping and then execute these plans via an obstacle-avoiding controller. The README emphasizes the separation between high-level planning (using a transformer-based, cross-modal planner) and low-level control (using a trial-and-error heuristic).

##### Major Components
a. Habitat Simulator & Extensions:
– The folder “habitat_extensions” contains customized modules (e.g., configuration, navigation, sensors, control, and observation transformers) that extend Facebook’s Habitat simulator.
– This module is responsible for interacting with the physical simulation environment (rendering scenes, processing sensor data, etc.).

b. Precomputed Visual Features Module:
– “precompute_img_features” includes scripts and connectivity files for extracting and storing image, depth, and semantic features.
– It interfaces with the simulation outputs to precompute features using visual encoders (e.g., the ResNet-based encoder).

c. Pre-training and Model Building:
– Directory “pretrain_src” holds model definitions, datasets, optimizers, and pretraining routines.
– It encapsulates the transformer models (language and visual feature encoders) as well as modules for constructing the cross-modal planner.
– Related configuration files and bash runner scripts (e.g., “run_pt/run_r2r.bash”) indicate the pretraining workflow.

d. Fine-tuning, Evaluation, and Inference:
– These modules are responsible for fitting and testing the VLN agent on the task datasets, bridging the pre-trained models with deployment in simulation.

e. Baselines and Auxiliary Tools:
– The “vlnce_baselines” directory contains additional baseline implementations, configuration files for training (including variations for different approaches), and various utility modules.
– It suggests alternative architectures or supporting experiments to compare against ETPNav’s performance.

f. External Configurations and Pre-trained Models:
– The “bert_config” folder holds pre-trained language model configurations and tokenizers (e.g., for BERT and XLM-Roberta), which are critical for the language understanding part of the cross-modal planning.
– Data folders (like “data/…” mentioned within the README) and the pretrained checkpoints (organized under “pretrained/ETP”) form part of the ecosystem.

##### Relationships and Interactions
- The simulation environment (via Habitat and its extensions) provides sensory inputs and a dynamic world state.
- Precomputed feature extraction scripts process these sensory inputs, hand off requisite visual features to the transformer-based planner.
- The planner (built using modules from pretrain_src) integrates language instructions (configured by the bert_config files) with visual features to generate navigation plans.
- The output of the high-level planner is then sent to the control module (within habitat_extensions) to execute low-level, obstacle-avoiding navigation.
- Feedback loops exist where sensor updates continuously inform the planner/control loop.
- Fine-tuning workflows and baseline comparisons are orchestrated through distinct run scripts to optimize model performance.

##### Architectural Patterns and Design Principles
– There is a clear separation of concerns: simulation and environment handling, feature extraction/preprocessing, model training (with pre-training and fine-tuning stages), and execution/control.
– The design utilizes modularity: different directories (habitat_extensions, pretrain_src, precompute_img_features, vlnce_baselines) encapsulate distinct responsibilities.
– Dependency injection is achieved through configuration files (YAML and JSON files in various config folders) that make the model and environment behavior customizable.
– The use of transformer architectures and precomputed features indicates an emphasis on cross-modal learning and efficient inference pipelines.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Environment & Simulation"
        sim["Simulation & Control"]
    end

    subgraph "Visual Feature Extraction"
        feat["Precomputed Visual Features"]
    end

    subgraph "Model Pre-training & Planning"
        pretrain["Pretrain & Model Building"]
        lang["Language Configs"]
    end

    subgraph "Fine-tuning and Inference"
        r2r["Run R2R"]
        rxr["Run RxR"]
    end

    subgraph "Baselines and Auxiliary Tools"
        baseline["Baselines Tools"]
    end

    sim -->|"sensory_data"| feat
    feat -->|"visual_features"| pretrain
    lang -->|"language_specs"| pretrain
    pretrain -->|"nav_plan"| sim
    sim -->|"sensor_update"| pretrain
    r2r -->|"finetune"| pretrain
    rxr -->|"finetune"| pretrain
    baseline -->|"compare"| pretrain

    class sim simulation
    class feat feature
    class pretrain planning
    class lang planning
    class r2r tuning
    class rxr tuning
    class baseline baseline

    classDef simulation fill:#e6f7ff,stroke:#333,stroke-width:2px;
    classDef feature fill:#e6ffe6,stroke:#333,stroke-width:2px;
    classDef planning fill:#ffe6e6,stroke:#333,stroke-width:2px;
    classDef tuning fill:#fff0b3,stroke:#333,stroke-width:2px;
    classDef baseline fill:#e0e0e0,stroke:#333,stroke-width:2px;

    click sim "https://github.com/marsaki/etpnav/tree/main/habitat_extensions/"
    click feat "https://github.com/marsaki/etpnav/tree/main/precompute_img_features/"
    click pretrain "https://github.com/marsaki/etpnav/tree/main/pretrain_src/"
    click lang "https://github.com/marsaki/etpnav/tree/main/bert_config/"
    click r2r "https://github.com/marsaki/etpnav/tree/main/run_r2r/"
    click rxr "https://github.com/marsaki/etpnav/tree/main/run_rxr/"
    click baseline "https://github.com/marsaki/etpnav/tree/main/vlnce_baselines/"
```
