How yonggie/navvlm works
Project Type and Purpose
• This is a full-stack Python research project for embodied navigation in simulated 3D environments.
• It orchestrates a Habitat-Sim environment, runs detection and segmentation (Detic, Grounded-SAM), feeds perceptions into a vision-language model, maintains instance memory, and plans navigation actions.
File Structure
• Top-level scripts:
• NavVLM.py (entrypoint / orchestrator)
• goat_matching.py, superglue.py (auxiliary matching routines)
• load_sim_utils.py (Habitat-Sim loader)
• Core interfaces & base classes in core/:
• abstract_perception.py, interfaces.py
• Perception stack in perception/:
• detection (Detic, Grounded-SAM)
• utilities for tokenization, sampler
• Configuration in navi_config.py & constants.py
• Memory module: instance_memory.py
• Model definition in model.py
• Planning & geometry in utils/:
• fmm_planner.py, depth_utils, point_cloud, pose, rotation, visualization
• Third-party pretrained nets under third_party/
README Insights
• Uses Habitat-Sim with Bullet physics
• Integrates Detectron2, CLIP, CenterNet2 submodule
• Main command: python NavVLM.py
Main Components & Their Interactions
a. Components
1) Entrypoint & Config
– NavVLM.py reads navi_config.py, constants.py
2) Simulator Environment
– load_sim_utils.py uses Habitat-Sim to load datasets (HM3D, Gibson, MP3D) and .glb scenes
3) Perception Subsystem
– Implements abstract_perception interface
– Detecion via DETIC (perception/detection/detic) + CLIP tokenizer
– Grounded-SAM for segmentation
4) Vision-Language Model (model.py)
– Combines detection outputs + language embedding via CLIP
5) Instance Memory (instance_memory.py)
– Tracks object instances over time
6) Planner (utils/fmm_planner.py)
– Fast marching method grid-based path planning
7) Matching & Tracking Extras
– SuperGluePretrainedNetwork (third_party)
– goat_matching.py, superglue.py
8) Utilities
– Depth, geometry, visualization, distributions, morphology, spot robot API
b. Data Flows & Relationships
1) NavVLM.py → loads config
2) initializes Habitat-Sim (load_sim_utils) → returns environment handle
3) Each timestep:
• RGB-D & point cloud → feed to Perception modules
• Vision-Language Model ingests perception + language goal → outputs next action
• Planner uses current pose + global map → refines path
• Action sent to simulator → new state
4) Visualization utilities render intermediate results
Architectural Patterns & Principles
• Layered architecture:
• Orchestrator layer (NavVLM.py)
• Abstraction layer (core/interfaces)
• Perception layer (detection, segmentation)
• Reasoning layer (model, memory, planning)
• Infrastructure layer (utils, Habitat-Sim)
• Dependency injection via interfaces in core/ for swapping perception backends
• Modular design: each subsystem in its own directory
• Use of third-party pretrained modules follows “external service” pattern
Technologies, Frameworks & Libraries
• Python 3.9
• Habitat-Sim & Habitat-Lab
• Detectron2, CenterNet2, DETIC
• OpenAI CLIP
• PyTorch (underlying for DETIC, SuperGlue)
• NumPy, SciPy for planning & geometry