How morigakuto/uiap-ogn works
Project Type and Purpose
• This is a robotics research codebase implementing “Uncertainty-Informed Active Perception for Open Vocabulary Object Goal Navigation.” It integrates a simulation environment (Habitat), vision-language models (VLMs) served via Flask, a probabilistic sensor model and mapping module, and a frontier exploration planner guided by semantic uncertainty. The primary goal is to autonomously explore indoor 3D environments to locate objects specified in natural language.
High-Level Components
A. Entry Point & Orchestration
• vlfm/run.py: main script that kicks off evaluation or training loops.
• scripts/launch_vlm_servers.sh: spins up VLM inference servers (GroundingDINO, BLIP2, etc.) in TMUX.
B. Perception & VLM Servers
• vlfm/vlm/: wrappers around VLM backends (BLIP2, GroundingDINO, YOLOv7, SAM) exposed via Flask (server_wrapper.py).
• External: GroundingDINO and YOLOv7 repositories (cloned under root).
C. Observation Processing
• vlfm/obs_transformers/: resizes or preprocesses raw RGB/depth frames.
• vlfm/utils/img_utils.py, habitat_visualizer.py: image formatting and debugging.
D. Probabilistic Sensor Model & Mapping
• vlfm/mapping/:
• base_map.py: abstract Map interface
• obstacle_map.py, frontier_map.py, value_map.py, object_point_cloud_map.py, vlplan_context_map.py
• traj_visualizer.py: map visualization helper
E. Exploration Planner & Policies
• vlfm/policy/:
• BasePolicy classes, objectnav-specific policies, ITM-based policies, reality-based policies
• vlplan_itm_policy.py: the uncertainty-informed multi-armed bandit planner
• vlfm/utils/vlfm_trainer.py: orchestrates training loop if applicable
F. Simulation & Reality Interface
• vlfm/reality/: Habitat-based environments and robot abstractions
• objectnav_env.py, pointnav_env.py, bdsw_nav_env.py
• base_robot.py, bdsw_robot.py (Boston Dynamics Spot)
• run_bdsw_objnav_env.py
G. Configuration, Data & Scripts
• config/: YAML files defining experiments and tasks (semantic task, pointnav depth, etc.)
• data/: pretrained weights (pointnav, SAM, YOLOv7, etc.)
• docker/: Dockerfile for containerization
• scripts/: evaluation wrappers (eval_*.sh), JSON parsers, VLM server launcher
Interactions & Data Flow
1. Initialization:
• Run vlfm/run.py (with configs from config/experiments/*.yaml).
• Launch VLM servers via scripts/launch_vlm_servers.sh.
2. At each simulation step:
a. Habitat sim (vlfm/reality) emits RGB/depth frames.
b. Frames → obs_transformers → sent to VLM Flask endpoints.
c. VLM servers return object detections + semantic embeddings + uncertainty estimates.
d. Sensor model (vlfm/mapping) ingests detections, updates probabilistic geometric-semantic map (value_map, frontier_map, object_point_cloud_map).
e. Planner (vlplan_itm_policy) uses map + multi-armed bandit objective to select next frontier/location to explore.
f. Policy module converts that decision into low-level robot actions (turn, forward, etc.).
g. Actions issued back to Habitat sim → new pose → loop.
3. Logging & Metrics:
• vlfm/utils/episode_stats_logger.py collects success, SPL, path length.
• scripts/parse_jsons.py aggregates evaluation outputs.
Key Architectural Patterns
• Modular plug-in design for perception backends (each VLM served via Flask)
• Clear separation of concerns: Perception ↔ Mapping ↔ Planning ↔ Action Execution
• Probabilistic mapping pattern: maintain belief maps (value_map, uncertainty_map)
• Frontier exploration orchestrated by an uncertainty-informed multi-armed bandit
Technologies and Frameworks
• Python 3.9, PyTorch (1.12.1+cu113), Habitat-lab & Habitat-sim
• Vision-Language: GroundingDINO, BLIP2, YOLOv7, MobileSAM
• Flask microservices for VLM inference
• TMUX for model server management
• YAML for experiment/task configuration
• Docker for reproducible environments