# anjiecheng/navila architecture

> Architecture diagram of the GitHub repository anjiecheng/navila, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anjiecheng/navila
- Repository: https://github.com/anjiecheng/navila

## Overview

##### Project type and purpose
- NaVILA is a research‐oriented navigation framework that fuses Vision–Language–Action (VLA) modeling with real‐time legged‐robot locomotion.
- It provides:
- A multimodal training pipeline (based on LLAVA/VILA architectures) for generating high‐level language commands.
- A two‐level control loop: (a) a language‐driven planner, (b) a real‐time locomotion policy for obstacle avoidance.
- An evaluation suite built on Habitat‐Lab/Sim and VLN‐CE to benchmark navigation tasks (R2R, RxR, ScanQA, YouTube Human Touring).

##### File structure
Top‐level directories:
- evaluation/ — evaluation harness: Habitat extensions, VLN‐CE baselines, scripts to run and aggregate metrics
- llava/          — core multimodal library: data ingestion, model definitions, training and evaluation CLI
- scripts/        — utility scripts for data extraction, launch training
- assets/, environment_setup.sh, pyproject.toml, README.md
Patterns and choices:
- “llava/data” & “llava/data_aug”: dataset builders & augmentation
- “llava/model”: modular design (language model + multimodal encoder + projector + loss)
- “llava/train”: multi‐GPU/distributed training (Deepspeed, sequence‐parallel, SLURM)
- “llava/trl”: reinforcement‐learning/trainer abstractions (PPO, DPO, SFT, etc.)
- evaluation/vlnce_baselines: imitation/RL baselines for command following in Habitat

##### README for additional insights
- Two workflows:
a. Training (setup conda, download external VQA and VLN datasets, run scripts/train/sft_8frames.sh)
b. Evaluation (build Habitat‐Sim/Lab v0.1.7, install dependencies, run evaluation/scripts/r2r.sh, aggregate metrics)
- External services & tools: Hugging Face (model checkpoints, datasets), Habitat‐CE, FlashAttention, deepspeed, Transformers v4.37.2 hacks

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources & Preprocessing
    subgraph "Data Sources & Preprocessing"
        DS1["R2R/RxR Datasets"]:::data
        DS2["Video QA Datasets"]:::data
        ER["Extract raw frames"]:::data
        D1["llava/data"]:::data
        DA["llava/data_aug"]:::data
    end
    DS1 --> ER
    DS2 --> ER
    ER --> D1
    ER --> DA

    %% Model Architecture
    subgraph "Model Architecture"
        ME["Multimodal Encoder"]:::model
        MP["Multimodal Projector"]:::model
        LM["Language Model"]:::model
        LA["LLAVA Fusion & Arch"]:::model
        LO["Loss Computation"]:::model
    end

    %% Training Pipeline & Orchestration
    subgraph "Training Pipeline & Orchestration"
        T1["train.py"]:::model
        T2["train_hybrid.py"]:::model
        T3["train_llm_to_long.py"]:::model
        T4["train_long.py"]:::model
        T5["llava_trainer.py"]:::model
        SLURM["slurm_utils.py"]:::model
        SEQ["sequence_parallel/"]:::model
        DSREP["deepspeed_replace/"]:::model
        SH["sft_8frames.sh"]:::model
    end

    %% Reinforcement-Learning & SFT Trainers
    subgraph "Reinforcement-Learning & SFT Trainers"
        PPO["PPO Trainer"]:::model
        DPO["DPO Trainer"]:::model
        SFT["SFT Trainer"]:::model
        DDPO["DDPO Trainer"]:::model
        ITER["Iterative SFT Trainer"]:::model
    end

    %% Checkpoint
    CKPT["Model Checkpoint"]:::model

    %% Inference & CLI
    subgraph "Inference & CLI"
        ENTRY["entry.py"]:::model
        RUN["llava/cli/run.py"]:::model
        EVALCLI["llava/cli/eval.py"]:::model
    end

    %% Evaluation Suite
    subgraph "Evaluation Suite"
        subgraph "Habitat Extensions"
            HAB_EXT["evaluation/habitat_extensions"]:::eval
        end
        subgraph "VLN-CE Baselines"
            VLN_BASE["evaluation/vlnce_baselines"]:::eval
        end
        subgraph "Eval Scripts"
            EVAL_S["evaluation/scripts"]:::eval
        end
        MET["Metrics Aggregation"]:::eval
    end

    %% Environment Setup
    subgraph "Environment & Setup"
        ENV["environment_setup.sh"]:::ext
        PY["pyproject.toml"]:::ext
        REQ["evaluation/requirements.txt"]:::ext
        README["README.md"]:::ext
    end

    %% External Dependencies
    subgraph "External Dependencies"
        HF["Hugging Face"]:::ext
        DSERV["Deepspeed/FlashAttention"]:::ext
        HABSIM["Habitat-Sim/Lab"]:::ext
    end

    %% Data flow connections
    D1 & DA --> ME
    ME --> MP
    MP --> LM
    LM --> LA
    LA --> LO
    LO --> T1
    LO --> T2
    LO --> T3
    LO --> T4
    LO --> T5
    LO --> SLURM
    LO --> SEQ
    LO --> DSREP
    LO --> SH

    %% Training to checkpoint
    T1 & T2 & T3 & T4 & T5 & SLURM & SEQ & DSREP & SH --> CKPT

    %% RL/SFT Trainers also feed checkpoint
    CKPT --> PPO
    CKPT --> DPO
    CKPT --> SFT
    CKPT --> DDPO
    CKPT --> ITER
    PPO & DPO & SFT & DDPO & ITER --> CKPT

    %% Checkpoint to inference & evaluation
    CKPT --> ENTRY
    CKPT --> RUN
    CKPT --> EVALCLI
    CKPT --> HAB_EXT
    CKPT --> VLN_BASE
    HAB_EXT --> EVAL_S
    VLN_BASE --> EVAL_S
    EVAL_S --> MET

    %% Environment & external
    ENV --> D1
    ENV --> LO
    ENV --> EVAL_S
    PY --> LO
    REQ --> HAB_EXT
    README --> D1

    HF --> ME
    HF --> LM
    DSERV --> LO
    DSERV --> PPO
    DSERV --> DPO
    HABSIM --> HAB_EXT

    %% Click Events
    click ER "https://github.com/anjiecheng/navila/blob/main/scripts/extract_rawframes.py"
    click D1 "https://github.com/anjiecheng/navila/tree/main/llava/data/"
    click DA "https://github.com/anjiecheng/navila/tree/main/llava/data_aug/"
    click ME "https://github.com/anjiecheng/navila/tree/main/llava/model/multimodal_encoder/"
    click MP "https://github.com/anjiecheng/navila/tree/main/llava/model/multimodal_projector/"
    click LM "https://github.com/anjiecheng/navila/tree/main/llava/model/language_model/"
    click LA "https://github.com/anjiecheng/navila/blob/main/llava/model/llava_arch.py"
    click LO "https://github.com/anjiecheng/navila/blob/main/llava/model/loss.py"
    click T1 "https://github.com/anjiecheng/navila/blob/main/llava/train/train.py"
    click T2 "https://github.com/anjiecheng/navila/blob/main/llava/train/train_hybrid.py"
    click T3 "https://github.com/anjiecheng/navila/blob/main/llava/train/train_llm_to_long.py"
    click T4 "https://github.com/anjiecheng/navila/blob/main/llava/train/train_long.py"
    click T5 "https://github.com/anjiecheng/navila/blob/main/llava/train/llava_trainer.py"
    click SLURM "https://github.com/anjiecheng/navila/blob/main/llava/train/slurm_utils.py"
    click SEQ "https://github.com/anjiecheng/navila/tree/main/llava/train/sequence_parallel/"
    click DSREP "https://github.com/anjiecheng/navila/tree/main/llava/train/deepspeed_replace/"
    click SH "https://github.com/anjiecheng/navila/blob/main/scripts/train/sft_8frames.sh"
    click PPO "https://github.com/anjiecheng/navila/blob/main/llava/trl/trainer/ppo_trainer.py"
    click DPO "https://github.com/anjiecheng/navila/blob/main/llava/trl/trainer/dpo_trainer.py"
    click SFT "https://github.com/anjiecheng/navila/blob/main/llava/trl/trainer/sft_trainer.py"
    click DDPO "https://github.com/anjiecheng/navila/blob/main/llava/trl/trainer/ddpo_trainer.py"
    click ITER "https://github.com/anjiecheng/navila/blob/main/llava/trl/trainer/iterative_sft_trainer.py"
    click ENTRY "https://github.com/anjiecheng/navila/blob/main/llava/entry.py"
    click RUN "https://github.com/anjiecheng/navila/blob/main/llava/cli/run.py"
    click EVALCLI "https://github.com/anjiecheng/navila/blob/main/llava/cli/eval.py"
    click HAB_EXT "https://github.com/anjiecheng/navila/tree/main/evaluation/habitat_extensions/"
    click VLN_BASE "https://github.com/anjiecheng/navila/tree/main/evaluation/vlnce_baselines/"
    click EVAL_S "https://github.com/anjiecheng/navila/tree/main/evaluation/scripts/"
    click ENV "https://github.com/anjiecheng/navila/blob/main/environment_setup.sh"
    click PY "https://github.com/anjiecheng/navila/blob/main/pyproject.toml"
    click REQ "https://github.com/anjiecheng/navila/blob/main/evaluation/requirements.txt"
    click README "https://github.com/anjiecheng/navila/blob/main/README.md"

    %% Styles
    classDef data fill:#cfe2f3,stroke:#6fa8dc
    classDef model fill:#d9ead3,stroke:#6aa84f
    classDef eval fill:#fce5cd,stroke:#e69138
    classDef ext fill:#d9d2e9,stroke:#8e7cc3
```
