# yonggie/navvlm architecture

> Architecture diagram of the GitHub repository yonggie/navvlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yonggie/navvlm
- Repository: https://github.com/yonggie/navvlm

## Overview

##### Project Type and Purpose
- This is a full-stack Python research project for embodied navigation in simulated 3D environments.
- It orchestrates a Habitat-Sim environment, runs detection and segmentation (Detic, Grounded-SAM), feeds perceptions into a vision-language model, maintains instance memory, and plans navigation actions.

##### File Structure
- Top-level scripts:
- NavVLM.py (entrypoint / orchestrator)
- goat_matching.py, superglue.py (auxiliary matching routines)
- load_sim_utils.py (Habitat-Sim loader)
- Core interfaces & base classes in `core/`:
- abstract_perception.py, interfaces.py
- Perception stack in `perception/`:
- detection (Detic, Grounded-SAM)
- utilities for tokenization, sampler
- Configuration in navi_config.py & constants.py
- Memory module: instance_memory.py
- Model definition in model.py
- Planning & geometry in `utils/`:
- fmm_planner.py, depth_utils, point_cloud, pose, rotation, visualization
- Third-party pretrained nets under `third_party/`

##### README Insights
- Uses Habitat-Sim with Bullet physics
- Integrates Detectron2, CLIP, CenterNet2 submodule
- Main command: python NavVLM.py

##### Main Components & Their Interactions
a. Components
1) Entrypoint & Config
– NavVLM.py reads navi_config.py, constants.py
2) Simulator Environment
– load_sim_utils.py uses Habitat-Sim to load datasets (HM3D, Gibson, MP3D) and .glb scenes
3) Perception Subsystem
– Implements abstract_perception interface
– Detecion via DETIC (perception/detection/detic) + CLIP tokenizer
– Grounded-SAM for segmentation
4) Vision-Language Model (model.py)
– Combines detection outputs + language embedding via CLIP
5) Instance Memory (instance_memory.py)
– Tracks object instances over time
6) Planner (utils/fmm_planner.py)
– Fast marching method grid-based path planning
7) Matching & Tracking Extras
– SuperGluePretrainedNetwork (third_party)
– goat_matching.py, superglue.py
8) Utilities
– Depth, geometry, visualization, distributions, morphology, spot robot API

b. Data Flows & Relationships
1) NavVLM.py → loads config
2) initializes Habitat-Sim (load_sim_utils) → returns environment handle
3) Each timestep:
- RGB-D & point cloud → feed to Perception modules
- Vision-Language Model ingests perception + language goal → outputs next action
- Planner uses current pose + global map → refines path
- Action sent to simulator → new state
4) Visualization utilities render intermediate results

##### Architectural Patterns & Principles
- Layered architecture:
- Orchestrator layer (NavVLM.py)
- Abstraction layer (core/interfaces)
- Perception layer (detection, segmentation)
- Reasoning layer (model, memory, planning)
- Infrastructure layer (utils, Habitat-Sim)
- Dependency injection via interfaces in `core/` for swapping perception backends
- Modular design: each subsystem in its own directory
- Use of third-party pretrained modules follows “external service” pattern

##### Technologies, Frameworks & Libraries
- Python 3.9
- Habitat-Sim & Habitat-Lab
- Detectron2, CenterNet2, DETIC
- OpenAI CLIP
- PyTorch (underlying for DETIC, SuperGlue)
- NumPy, SciPy for planning & geometry

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers and components

    subgraph "Orchestrator Layer"
        direction TB
        NavVLM["NavVLM.py\n(Orchestrator)"]:::custom
        Config["navi_config.py\nconstants.py\n(Configuration)"]:::custom
    end

    subgraph "Abstraction Layer"
        direction TB
        Abstract["abstract_perception.py\n(Perception Interface)"]:::custom
        Interfaces["interfaces.py\n(Core Interfaces)"]:::custom
    end

    subgraph "Simulator Environment"
        direction TB
        Loader["load_sim_utils.py\n(Env Loader)"]:::custom
        Habitat["Habitat-Sim\n(External Simulator)"]:::external
        Scene["example_glb/Dd4bFSTQ8gi.glb\n(Scene Asset)"]:::external
    end

    subgraph "Perception Layer"
        direction TB
        PerceptionConst["perception/constants.py\n(Perception Constants)"]:::custom
        subgraph "Detection"
            direction TB
            Tokenizer["clip_simple_tokenizer.py\n(Tokenizer)"]:::custom
            DETICWrap["detic_perception.py\n(DETIC Wrapper)"]:::custom
            UtilsDet["utils.py\n(Detection Utils)"]:::util
        end
        Segmentation["grounded_sam\n(Grounded-SAM)"]:::external
    end

    subgraph "Reasoning Layer"
        direction TB
        Model["model.py\n(Vision-Language Model)"]:::custom
        Memory["instance_memory.py\n(Instance Memory)"]:::custom
        Planner["fmm_planner.py\n(Planner)"]:::custom
    end

    subgraph "Matching & Tracking Extras"
        direction TB
        GOAT["goat_matching.py\n(GOAT Matching)"]:::custom
        SuperGlueScript["superglue.py\n(SuperGlue Integration)"]:::custom
        SuperGlueNet["SuperGluePretrainedNetwork\n(Pretrained Net)"]:::external
    end

    subgraph "Utility Modules"
        direction TB
        Depth["depth.py\n(Depth Processing)"]:::util
        DepthUtils["depth_utils.py\n(Depth Utils)"]:::util
        PointCloud["point_cloud.py\n(Point Cloud)"]:::util
        Pose["pose.py\n(Pose Representation)"]:::util
        Rotation["rotation.py\n(Rotation)"]:::util
        RotationUtils["rotation_utils.py\n(Rotation Utils)"]:::util
        Geometry["_base.py\n(Geometry Primitives)"]:::util
        Visualization["visualization.py\n(Visualization)"]:::util
        Distributions["distributions.py\n(Distributions)"]:::util
        Morphology["morphology.py\n(Morphology)"]:::util
        Spot["spot.py\n(Spot Robot API)"]:::util
        ConstUtils["constants.py\n(General Constants)"]:::util
    end

    %% Connections
    NavVLM -->|loads| Config
    NavVLM -->|initializes| Loader
    Loader -->|loads scenes| Scene
    Loader -->|creates| Habitat

    Habitat -->|captures RGB-D| Tokenizer
    Habitat -->|captures RGB-D| DETICWrap
    Habitat -->|captures RGB-D| Segmentation

    Tokenizer --> DETICWrap
    DETICWrap -->|detections| UtilsDet
    UtilsDet -->|percepts| Memory
    Segmentation -->|masks| Memory

    Memory -->|instances| Model
    DETICWrap -->|features| Model
    Segmentation -->|features| Model

    Model -->|action logits| Planner
    Planner -->|path| Habitat
    Model -->|action| Habitat

    Habitat -->|state| NavVLM

    %% Matching & Tracking Extras connections
    GOAT -->|matches| Memory
    SuperGlueScript -->|matches| Memory
    SuperGlueNet -->|supports| SuperGlueScript

    %% Utilities usage
    Depth -->|processes| Habitat
    DepthUtils -->|supports| Depth
    PointCloud -->|generates| Model
    Pose -->|updates| Planner
    Rotation -->|helps| Planner
    RotationUtils -->|supports| Rotation
    Geometry -->|used by| Planner
    Visualization -->|renders| NavVLM
    Distributions -->|used by| Planner
    Morphology -->|used by| UtilsDet
    Spot -->|controls| Habitat
    ConstUtils -->|constants| Planner

    %% Layer dependencies
    NavVLM --> Abstract
    Abstract --> DETICWrap
    Abstract --> Segmentation

    %% Styles
    classDef custom fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef external fill:#90EE90,stroke:#333,stroke-width:1px
    classDef util fill:#D3D3D3,stroke:#333,stroke-width:1px

    %% Click Events
    click NavVLM "https://github.com/yonggie/navvlm/blob/main/NavVLM.py"
    click Config "https://github.com/yonggie/navvlm/blob/main/navi_config.py"
    click Config "https://github.com/yonggie/navvlm/blob/main/constants.py"
    click Loader "https://github.com/yonggie/navvlm/blob/main/load_sim_utils.py"
    click Scene "https://github.com/yonggie/navvlm/blob/main/example_glb/Dd4bFSTQ8gi.glb"
    click Abstract "https://github.com/yonggie/navvlm/blob/main/core/abstract_perception.py"
    click Interfaces "https://github.com/yonggie/navvlm/blob/main/core/interfaces.py"
    click PerceptionConst "https://github.com/yonggie/navvlm/blob/main/perception/constants.py"
    click Tokenizer "https://github.com/yonggie/navvlm/blob/main/perception/detection/detic/clip_simple_tokenizer.py"
    click DETICWrap "https://github.com/yonggie/navvlm/blob/main/perception/detection/detic/detic_perception.py"
    click Segmentation "https://github.com/yonggie/navvlm/tree/main/perception/detection/grounded_sam"
    click UtilsDet "https://github.com/yonggie/navvlm/blob/main/perception/detection/utils.py"
    click Model "https://github.com/yonggie/navvlm/blob/main/model.py"
    click Memory "https://github.com/yonggie/navvlm/blob/main/instance_memory.py"
    click Planner "https://github.com/yonggie/navvlm/blob/main/utils/fmm_planner.py"
    click GOAT "https://github.com/yonggie/navvlm/blob/main/goat_matching.py"
    click SuperGlueScript "https://github.com/yonggie/navvlm/blob/main/superglue.py"
    click SuperGlueNet "https://github.com/yonggie/navvlm/tree/main/third_party/SuperGluePretrainedNetwork"
    click Depth "https://github.com/yonggie/navvlm/blob/main/utils/depth.py"
    click DepthUtils "https://github.com/yonggie/navvlm/blob/main/utils/depth_utils.py"
    click PointCloud "https://github.com/yonggie/navvlm/blob/main/utils/point_cloud.py"
    click Pose "https://github.com/yonggie/navvlm/blob/main/utils/pose.py"
    click Rotation "https://github.com/yonggie/navvlm/blob/main/utils/rotation.py"
    click RotationUtils "https://github.com/yonggie/navvlm/blob/main/utils/rotation_utils.py"
    click Geometry "https://github.com/yonggie/navvlm/blob/main/utils/geometry/_base.py"
    click Visualization "https://github.com/yonggie/navvlm/blob/main/utils/visualization.py"
    click Distributions "https://github.com/yonggie/navvlm/blob/main/utils/distributions.py"
    click Morphology "https://github.com/yonggie/navvlm/blob/main/utils/morphology.py"
    click Spot "https://github.com/yonggie/navvlm/blob/main/utils/spot.py"
    click ConstUtils "https://github.com/yonggie/navvlm/blob/main/utils/constants.py"
```
