# h-embodvis/nautilus architecture

> Architecture diagram of the GitHub repository h-embodvis/nautilus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/h-embodvis/nautilus
- Repository: https://github.com/h-embodvis/nautilus

## Overview

##### Project type and purpose
- NAUTILUS is an open-source, research-grade Large Multimodal Model (LMM) framework for underwater scene understanding. It provides two fine-tuning and inference pipelines (LLaVA-based and Qwen-based), a plug-and-play Vision Feature Enhancement (VFE) module, evaluation utilities, and dataset preprocessing.

##### File structure
- Top-level directories:
- LLaVA: the core Python package implementing model components (image/vision encoder, depth encoder, VFE, vision-to-language projector, LLM interfaces), training scripts, evaluation modules, and a Gradio/CLI serving layer.
- qwen-vl-finetune: parallel code for fine-tuning and inference on Qwen2.5-VL, with its own data loader, model builder, training, and scripts.
- eval: generic evaluation scripts and an Evaluation_pack library for metric calculations (classification, detection, grounding, etc.).
- scripts: convenience shell and Python scripts for inference, merging LoRA weights, and multi-stage evaluations.
- utils: weight-conversion and preprocessing utilities (e.g., process_vitl_weight.py).

##### README for additional insights
- Two modes: LLaVA vs Qwen2.5-VL, each with its own conda env, dependency set, fine-tuning script, and inference CLI.
- External dependencies: pretrained LLaVA/Vicuna, Qwen models, Depth-Anything V2, FlashAttention.

##### Diagramming recommendations
- Use clear labels per component (e.g., “VFE Module,” “LLM Adapter: Qwen2.5-VL”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing"
        style DataLayer fill:#D0E7FF,stroke:#333,stroke-width:1px
        NautData[(NautData Dataset)]:::data
        PretrainedWeights[(Pretrained Weights)]:::data
        Preprocessing["process_vitl_weight.py"]:::data
    end

    %% Model Construction Layer
    subgraph "Model Construction"
        style ModelLayer fill:#DFFFD0,stroke:#333,stroke-width:1px
        ClipEncoder["CLIP Encoder"]:::model
        DinoLayers["DINOv2 Layers"]:::model
        DepthEncoder["Depth Encoder"]:::model
        VFE["VFE Module"]:::model
        Projector["Vision-to-Language Projector"]:::model
        subgraph "LLM Adapters"
            LlavaAdapters["LLaVA Adapters"]:::model
            QwenAdapter["Qwen2.5-VL Adapter"]:::model
        end
    end

    %% Training Pipelines
    subgraph "Training Pipelines"
        style TrainLayer fill:#FFEFD0,stroke:#333,stroke-width:1px
        LlavaTrain["LLaVA Training Pipeline"]:::train
        QwenTrain["Qwen Training Pipeline"]:::train
    end

    %% Model Registry
    subgraph "Model Registry & Checkpoints"
        style RegistryLayer fill:#FFCFE0,stroke:#333,stroke-width:1px
        Checkpoints[(Local .pth/.bin Files)]:::storage
        HuggingFace[(HuggingFace Hub)]:::storage
    end

    %% Serving & Inference
    subgraph "Serving & Inference"
        style ServeLayer fill:#FFE4B5,stroke:#333,stroke-width:1px
        LlavaServe["LLaVA Serve (CLI & Gradio)"]:::serve
        QwenServe["Qwen Inference CLI"]:::serve
        EndUser((End-User Queries)):::external
    end

    %% Evaluation & Logging
    subgraph "Evaluation & Logging"
        style EvalLayer fill:#E8D0FF,stroke:#333,stroke-width:1px
        EvalUtils["Evaluation_pack Modules"]:::eval
        EvalScripts["Eval Scripts & Shells"]:::eval
        Logs["Monitoring & Logs"]:::eval
    end

    %% Data Flow Connections
    NautData -->|provides data| Preprocessing
    PretrainedWeights -->|load weights| ClipEncoder
    Preprocessing -->|prepared data| ClipEncoder & DinoLayers & DepthEncoder
    ClipEncoder --> VFE
    DinoLayers --> VFE
    DepthEncoder --> VFE
    VFE --> Projector
    Projector --> LlavaAdapters
    Projector --> QwenAdapter

    %% Training Flow
    LlavaAdapters --> LlavaTrain
    QwenAdapter --> QwenTrain
    LlavaTrain -->|save| Checkpoints
    QwenTrain -->|save| Checkpoints

    %% Serving Flow
    Checkpoints --> LlavaServe
    Checkpoints --> QwenServe
    EndUser -->|query| LlavaServe
    EndUser -->|query| QwenServe

    %% Evaluation & Logging Flow
    Checkpoints --> EvalScripts
    LlavaServe --> Logs
    QwenServe --> Logs
    EvalScripts --> EvalUtils
    EvalUtils --> Logs

    %% Click Events
    click Preprocessing "https://github.com/h-embodvis/nautilus/blob/main/utils/process_vitl_weight.py"
    click ClipEncoder "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/model/multimodal_encoder/builder.py"
    click DinoLayers "https://github.com/h-embodvis/nautilus/tree/main/qwen-vl-finetune/qwenvl/nautilus_model/dinov2_layers/"
    click DepthEncoder "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/model/dinov2.py"
    click VFE "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/model/vfe_layer.py"
    click Projector "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/model/multimodal_projector/builder.py"
    click LlavaAdapters "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/model/language_model/llava_mistral.py"
    click QwenAdapter "https://github.com/h-embodvis/nautilus/blob/main/qwen-vl-finetune/qwenvl/nautilus_model/dinov2.py"
    click LlavaTrain "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/train/train.py"
    click QwenTrain "https://github.com/h-embodvis/nautilus/blob/main/qwen-vl-finetune/qwenvl/train/train_qwen.py"
    click LlavaServe "https://github.com/h-embodvis/nautilus/blob/main/LLaVA/llava/serve/cli.py"
    click QwenServe "https://github.com/h-embodvis/nautilus/blob/main/qwen-vl-finetune/scripts/inference.py"
    click EvalUtils "https://github.com/h-embodvis/nautilus/tree/main/eval/Evaluation_pack/"
    click EvalScripts "https://github.com/h-embodvis/nautilus/blob/main/eval/eval_nautilus_llava.sh"
    click Logs "https://github.com/h-embodvis/nautilus/blob/main/logs/llava_logs.json"

    %% Styles
    classDef data fill:#D0E7FF,stroke:#1E90FF,stroke-width:2px
    classDef model fill:#DFFFD0,stroke:#32CD32,stroke-width:2px
    classDef train fill:#FFEFD0,stroke:#FFA500,stroke-width:2px
    classDef storage fill:#FFCFE0,stroke:#FF1493,stroke-width:2px
    classDef serve fill:#FFE4B5,stroke:#DAA520,stroke-width:2px
    classDef eval fill:#E8D0FF,stroke:#8A2BE2,stroke-width:2px
    classDef external fill:#FFFFFF,stroke:#000000,stroke-width:1px,stroke-dasharray: 5 5
```
