How anjiecheng/navila works
Project type and purpose
• NaVILA is a research‐oriented navigation framework that fuses Vision–Language–Action (VLA) modeling with real‐time legged‐robot locomotion.
• It provides:
• A multimodal training pipeline (based on LLAVA/VILA architectures) for generating high‐level language commands.
• A two‐level control loop: (a) a language‐driven planner, (b) a real‐time locomotion policy for obstacle avoidance.
• An evaluation suite built on Habitat‐Lab/Sim and VLN‐CE to benchmark navigation tasks (R2R, RxR, ScanQA, YouTube Human Touring).
File structure
Top‐level directories:
• evaluation/ — evaluation harness: Habitat extensions, VLN‐CE baselines, scripts to run and aggregate metrics
• llava/ — core multimodal library: data ingestion, model definitions, training and evaluation CLI
• scripts/ — utility scripts for data extraction, launch training
• assets/, environment_setup.sh, pyproject.toml, README.md
Patterns and choices:
• “llava/data” & “llava/data_aug”: dataset builders & augmentation
• “llava/model”: modular design (language model + multimodal encoder + projector + loss)
• “llava/train”: multi‐GPU/distributed training (Deepspeed, sequence‐parallel, SLURM)
• “llava/trl”: reinforcement‐learning/trainer abstractions (PPO, DPO, SFT, etc.)
• evaluation/vlnce_baselines: imitation/RL baselines for command following in Habitat
README for additional insights
• Two workflows:
a. Training (setup conda, download external VQA and VLN datasets, run scripts/train/sft_8frames.sh)
b. Evaluation (build Habitat‐Sim/Lab v0.1.7, install dependencies, run evaluation/scripts/r2r.sh, aggregate metrics)
• External services & tools: Hugging Face (model checkpoints, datasets), Habitat‐CE, FlashAttention, deepspeed, Transformers v4.37.2 hacks