How h-embodvis/nautilus works
Project type and purpose
• NAUTILUS is an open-source, research-grade Large Multimodal Model (LMM) framework for underwater scene understanding. It provides two fine-tuning and inference pipelines (LLaVA-based and Qwen-based), a plug-and-play Vision Feature Enhancement (VFE) module, evaluation utilities, and dataset preprocessing.
File structure
• Top-level directories:
• LLaVA: the core Python package implementing model components (image/vision encoder, depth encoder, VFE, vision-to-language projector, LLM interfaces), training scripts, evaluation modules, and a Gradio/CLI serving layer.
• qwen-vl-finetune: parallel code for fine-tuning and inference on Qwen2.5-VL, with its own data loader, model builder, training, and scripts.
• eval: generic evaluation scripts and an Evaluation_pack library for metric calculations (classification, detection, grounding, etc.).
• scripts: convenience shell and Python scripts for inference, merging LoRA weights, and multi-stage evaluations.
• utils: weight-conversion and preprocessing utilities (e.g., process_vitl_weight.py).
README for additional insights
• Two modes: LLaVA vs Qwen2.5-VL, each with its own conda env, dependency set, fine-tuning script, and inference CLI.
• External dependencies: pretrained LLaVA/Vicuna, Qwen models, Depth-Anything V2, FlashAttention.
Diagramming recommendations
• Use clear labels per component (e.g., “VFE Module,” “LLM Adapter: Qwen2.5-VL”).