How prince687028/traveluav works
Main Components
• Simulation Environment:
• AirSim Simulator – represented by the airsim_plugin, which provides both server (AirVLNSimulatorServerTool.py) and client (AirVLNSimulatorClientTool.py) functionalities.
• External environments (Carla, closeloop, extra) that supply the simulation maps.
• UAV Vision-Language Model (LLaMA-UAV):
• Core model components reside in Model/LLaMA-UAV. This includes:
– The primary language model implementations (e.g., llamavid, language_model submodules).
– Multimodal components including the multimodal encoder and projector – key for combining visual and language modalities.
– Training scripts and utilities for model tuning (in llamavid/train and scripts/llm/train_uav_llm.sh).
• Auxiliary Computer Vision Modules:
• GroundingDINO integration, which can be found under src/model_wrapper/utils/GroundingDINO. This part handles object detection and vision grounding, contributing to scene analysis.
• Pipeline Orchestration and Evaluation:
• Simulation orchestration and experimental routines in src/vlnce_src (e.g., eval.py, dagger.py) and scripts (e.g., dagger_NYC.sh, eval.sh, metric.sh).
• Common utilities and configurations in src/common, utils, and configuration scripts.
• Data and Environment Setup:
• Dataset and environment download paths – as referenced in the README – for preparing both training data and simulation maps.
Relationships and Interactions
• The Simulation Environment module (AirSim server and external simulator maps) interacts with the Model via simulation control scripts and API clients. Data flows from the simulation (environment state or sensor data) to the model.
• The LLaMA-UAV model receives visual inputs and language instructions. Within its architecture, the language model and multimodal encoder project features into a common space, using components from both the “llamavid” package and the “GroundingDINO” module.
• Pipeline orchestration layers coordinate the data flow:
• Preprocessing scripts and utility modules convert simulation data for model consumption.
• Training scripts (and associated configurations) trigger the model’s forward and backward passes, while evaluation scripts (in src/vlnce_src and scripts) manage inference and metric calculation.
• The GroundingDINO module supports the main model by providing visual grounding information which augments the decision-making process in the UAV navigation tasks.
Architectural Patterns and Design Principles
• Modularity: Each major function (simulation, model training/inference, vision-processing) is organized into its own directory and module. This enables independent development and testing.
• Plug-and-Play Components: Emphasis on how external modules (like AirSim APIs and GroundingDINO) integrate with the core navigation algorithm, promoting extensibility of the platform.
Relevant Technologies, Frameworks, and Libraries
• PyTorch (for model training/inference as installed in the environment setup).
• Conda and pip for environment management and dependency installation.
• Docker and configuration files (e.g., Dockerfile for GroundingDINO) that support deployment.
• Shell scripts for execution, training, and evaluation.