# morigakuto/uiap-ogn architecture

> Architecture diagram of the GitHub repository morigakuto/uiap-ogn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/morigakuto/uiap-ogn
- Repository: https://github.com/morigakuto/uiap-ogn

## Overview

##### Project Type and Purpose
- This is a robotics research codebase implementing “Uncertainty-Informed Active Perception for Open Vocabulary Object Goal Navigation.” It integrates a simulation environment (Habitat), vision-language models (VLMs) served via Flask, a probabilistic sensor model and mapping module, and a frontier exploration planner guided by semantic uncertainty. The primary goal is to autonomously explore indoor 3D environments to locate objects specified in natural language.

##### High-Level Components
A. Entry Point & Orchestration
- vlfm/run.py: main script that kicks off evaluation or training loops.
- scripts/launch_vlm_servers.sh: spins up VLM inference servers (GroundingDINO, BLIP2, etc.) in TMUX.

B. Perception & VLM Servers
- vlfm/vlm/: wrappers around VLM backends (BLIP2, GroundingDINO, YOLOv7, SAM) exposed via Flask (server_wrapper.py).
- External: GroundingDINO and YOLOv7 repositories (cloned under root).

C. Observation Processing
- vlfm/obs_transformers/: resizes or preprocesses raw RGB/depth frames.
- vlfm/utils/img_utils.py, habitat_visualizer.py: image formatting and debugging.

D. Probabilistic Sensor Model & Mapping
- vlfm/mapping/:
- base_map.py: abstract Map interface
- obstacle_map.py, frontier_map.py, value_map.py, object_point_cloud_map.py, vlplan_context_map.py
- traj_visualizer.py: map visualization helper

E. Exploration Planner & Policies
- vlfm/policy/:
- BasePolicy classes, objectnav-specific policies, ITM-based policies, reality-based policies
- vlplan_itm_policy.py: the uncertainty-informed multi-armed bandit planner
- vlfm/utils/vlfm_trainer.py: orchestrates training loop if applicable

F. Simulation & Reality Interface
- vlfm/reality/: Habitat-based environments and robot abstractions
- objectnav_env.py, pointnav_env.py, bdsw_nav_env.py
- base_robot.py, bdsw_robot.py (Boston Dynamics Spot)
- run_bdsw_objnav_env.py

G. Configuration, Data & Scripts
- config/: YAML files defining experiments and tasks (semantic task, pointnav depth, etc.)
- data/: pretrained weights (pointnav, SAM, YOLOv7, etc.)
- docker/: Dockerfile for containerization
- scripts/: evaluation wrappers (eval_*.sh), JSON parsers, VLM server launcher

##### Interactions & Data Flow
1. Initialization:
- Run `vlfm/run.py` (with configs from config/experiments/*.yaml).
- Launch VLM servers via `scripts/launch_vlm_servers.sh`.
2. At each simulation step:
a. Habitat sim (vlfm/reality) emits RGB/depth frames.
b. Frames → obs_transformers → sent to VLM Flask endpoints.
c. VLM servers return object detections + semantic embeddings + uncertainty estimates.
d. Sensor model (vlfm/mapping) ingests detections, updates probabilistic geometric-semantic map (value_map, frontier_map, object_point_cloud_map).
e. Planner (vlplan_itm_policy) uses map + multi-armed bandit objective to select next frontier/location to explore.
f. Policy module converts that decision into low-level robot actions (turn, forward, etc.).
g. Actions issued back to Habitat sim → new pose → loop.
3. Logging & Metrics:
- vlfm/utils/episode_stats_logger.py collects success, SPL, path length.
- scripts/parse_jsons.py aggregates evaluation outputs.

##### Key Architectural Patterns
- Modular plug-in design for perception backends (each VLM served via Flask)
- Clear separation of concerns: Perception ↔ Mapping ↔ Planning ↔ Action Execution
- Probabilistic mapping pattern: maintain belief maps (value_map, uncertainty_map)
- Frontier exploration orchestrated by an uncertainty-informed multi-armed bandit

##### Technologies and Frameworks
- Python 3.9, PyTorch (1.12.1+cu113), Habitat-lab & Habitat-sim
- Vision-Language: GroundingDINO, BLIP2, YOLOv7, MobileSAM
- Flask microservices for VLM inference
- TMUX for model server management
- YAML for experiment/task configuration
- Docker for reproducible environments

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry & Config
    Run["vlfm/run.py"]:::entry
    Launch["scripts/launch_vlm_servers.sh"]:::entry
    Config["Experiment Configs (YAML)"]:::config
    ConfigTasks["Task Configs (YAML)"]:::config
    DataWeights["Pretrained Weights"]:::data
    Docker["docker/Dockerfile"]:::script
    ParseJSON["scripts/parse_jsons.py"]:::script

    %% Simulation & Reality Interface
    subgraph "Simulation & Reality Interface"
        direction TB
        ObjectNavEnv["objectnav_env.py"]:::simulation
        PointNavEnv["pointnav_env.py"]:::simulation
        BDSWEnv["bdsw_nav_env.py"]:::simulation
        RunBDSW["run_bdsw_objnav_env.py"]:::simulation
        RobotBase["base_robot.py"]:::simulation
        RobotBDSW["bdsw_robot.py"]:::simulation
        CameraIDs["camera_ids.py"]:::simulation
        FrameIDs["frame_ids.py"]:::simulation
    end

    %% Perception & VLM Servers
    subgraph "Perception Pipeline"
        direction TB
        ObsTrans["obs_transformers"]:::perception
        ImgUtils["img_utils.py"]:::perception
        HabVis["habitat_visualizer.py"]:::perception
        subgraph "VLM Server Wrapper"
            direction TB
            ServerWrapper["server_wrapper.py"]:::perception
            BLIP2["blip2.py"]:::perception
            GroundDINO["grounding_dino.py"]:::perception
            YOLOv7["yolov7.py"]:::perception
            SAM["sam.py"]:::perception
        end
    end

    %% Mapping
    subgraph "Probabilistic Mapping"
        direction TB
        BaseMap["base_map.py"]:::mapping
        ObstacleMap["obstacle_map.py"]:::mapping
        FrontierMap["frontier_map.py"]:::mapping
        ValueMap["value_map.py"]:::mapping
        ObjPCMap["object_point_cloud_map.py"]:::mapping
        ContextMap["vlplan_context_map.py"]:::mapping
        TrajVis["traj_visualizer.py"]:::mapping
    end

    %% Planning & Policies
    subgraph "Exploration Planner & Policies"
        direction TB
        ITMPolicy["vlplan_itm_policy.py"]:::planning
        BasePolicy["base_policy.py"]:::planning
        BaseObjPolicy["base_objectnav_policy.py"]:::planning
        ITMVariants["itm_policy.py"]:::planning
        RealityPolicies["reality_policies.py"]:::planning
        HabitatPolicies["habitat_policies.py"]:::planning
        Trainer["vlfm_trainer.py"]:::planning
    end

    %% Data Stores & Scripts
    Config --> Run
    ConfigTasks --> Run
    DataWeights --> Run
    Docker --> Run
    Launch --> ServerWrapper
    Run --> ObjectNavEnv
    Run --> Trainer
    Run --> ParseJSON

    %% Simulation to Perception
    ObjectNavEnv -->|"RGB/Depth frames"| ObsTrans
    PointNavEnv -->|"RGB/Depth frames"| ObsTrans
    BDSWEnv -->|"RGB/Depth frames"| ObsTrans
    ObsTrans --> ImgUtils
    ImgUtils --> HabVis

    %% Perception to Mapping
    ObsTrans -->|"Preprocessed frames"| ServerWrapper
    ServerWrapper -->|"Detections + embeddings + uncertainty"| BaseMap
    ServerWrapper --> BaseMap

    %% Mapping internals flow
    BaseMap --> ObstacleMap
    BaseMap --> FrontierMap
    BaseMap --> ValueMap
    BaseMap --> ObjPCMap
    BaseMap --> ContextMap

    %% Mapping to Planning
    ContextMap -->|"Belief maps, uncertainty"| ITMPolicy
    ValueMap --> ITMPolicy

    %% Planning to Simulation (Actions)
    ITMPolicy -->|"Next action"| ObjectNavEnv
    ITMPolicy --> PointNavEnv
    ITMPolicy --> BDSWEnv

    %% Scripts and Logging
    ParseJSON -->|"Aggregates metrics"| Run

    %% Click Events
    click Run "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/run.py"
    click Launch "https://github.com/morigakuto/uiap-ogn/blob/main/scripts/launch_vlm_servers.sh"
    click ServerWrapper "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/vlm/server_wrapper.py"
    click BLIP2 "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/vlm/blip2.py"
    click GroundDINO "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/vlm/grounding_dino.py"
    click YOLOv7 "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/vlm/yolov7.py"
    click SAM "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/vlm/sam.py"
    click ObsTrans "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/obs_transformers/resize.py"
    click ObsTrans "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/obs_transformers/utils.py"
    click ImgUtils "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/utils/img_utils.py"
    click HabVis "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/utils/habitat_visualizer.py"
    click BaseMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/base_map.py"
    click ObstacleMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/obstacle_map.py"
    click FrontierMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/frontier_map.py"
    click ValueMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/value_map.py"
    click ObjPCMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/object_point_cloud_map.py"
    click ContextMap "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/vlplan_context_map.py"
    click TrajVis "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/mapping/traj_visualizer.py"
    click ITMPolicy "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/vlplan_itm_policy.py"
    click BasePolicy "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/base_policy.py"
    click BaseObjPolicy "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/base_objectnav_policy.py"
    click ITMVariants "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/itm_policy.py"
    click RealityPolicies "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/reality_policies.py"
    click HabitatPolicies "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/policy/habitat_policies.py"
    click Trainer "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/utils/vlfm_trainer.py"
    click ObjectNavEnv "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/objectnav_env.py"
    click PointNavEnv "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/pointnav_env.py"
    click BDSWEnv "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/bdsw_nav_env.py"
    click RunBDSW "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/run_bdsw_objnav_env.py"
    click RobotBase "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/robots/base_robot.py"
    click RobotBDSW "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/robots/bdsw_robot.py"
    click CameraIDs "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/robots/camera_ids.py"
    click FrameIDs "https://github.com/morigakuto/uiap-ogn/blob/main/vlfm/reality/robots/frame_ids.py"
    click Config "https://github.com/morigakuto/uiap-ogn/tree/main/config/experiments/"
    click ConfigTasks "https://github.com/morigakuto/uiap-ogn/tree/main/config/tasks/"
    click DataWeights "https://github.com/morigakuto/uiap-ogn/blob/main/data/pointnav_weights.pth"
    click Docker "https://github.com/morigakuto/uiap-ogn/tree/main/docker/Dockerfile"
    click ParseJSON "https://github.com/morigakuto/uiap-ogn/blob/main/scripts/parse_jsons.py"

    %% Styles
    classDef entry fill:#f9f,stroke:#333,stroke-width:1px
    classDef config fill:#ff9,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef data fill:#9ff,stroke:#333,stroke-width:1px
    classDef script fill:#fcc,stroke:#333,stroke-width:1px
    classDef perception fill:#bbdefb,stroke:#0d47a1,stroke-width:2px
    classDef mapping fill:#c8e6c9,stroke:#1b5e20,stroke-width:2px
    classDef planning fill:#ffe0b2,stroke:#e65100,stroke-width:2px
    classDef simulation fill:#e1bee7,stroke:#4a148c,stroke-width:2px
```
