# llava-vl/llava-next architecture

> Architecture diagram of the GitHub repository llava-vl/llava-next, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/llava-vl/llava-next
- Repository: https://github.com/llava-vl/llava-next

## Overview

LLaVA-NeXT is a multimodal vision-language model project supporting image, video, multi-image/interleaved inference, evaluation, and fine-tuning; the release notes also describe LLaVA-Critic-R1 training with GRPO. The main flow is user input through serving or evaluation, image/video preprocessing, multimodal model inference, and a generated answer. Separate branches represent supervised/DPO training and EasyR1 distributed RL training, plus the bundled TRL training library. Connections reflect README statements and SOURCE INDEX references; unsampled modules are represented only where the tree or documented capabilities supports them.

Main entry points: [`llava-critic-r1/EasyR1/verl/trainer/main.py`](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/trainer/main.py) (RL runner), [`llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py`](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py) (FSDP actor workers), [`llava/model/llava_arch.py`](https://github.com/llava-vl/llava-next/blob/main/llava/model/llava_arch.py) (LLaVA architecture), [`llava/model/builder.py`](https://github.com/llava-vl/llava-next/blob/main/llava/model/builder.py) (Model loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_multimodal["Multimodal Model"]
  node_mm_utils["Media preprocessing<br/>[mm_utils.py]"]
  node_vision["Vision encoders"]
  node_projector["Multimodal projector"]
  node_resampler["Visual resamplers"]
  node_llava_arch["LLaVA architecture<br/>[llava_arch.py]"]
  node_model_builder["Model loader<br/>[builder.py]"]
  node_language_models["Language model variants"]
end

subgraph group_inference["Inference and Evaluation"]
  node_serve_worker["Model worker<br/>[model_worker.py]"]
  node_serve_controller["Serving controller<br/>[controller.py]"]
  node_serve_clients["Chat clients<br/>[cli.py]"]
  node_evaluation["Interleave evaluation"]
end

subgraph group_training["Training"]
  node_training_entry["DPO training<br/>[train_dpo.py]"]
  node_llava_trainer["LLaVA trainer<br/>[llava_trainer.py]"]
end

subgraph group_rl["Distributed RL"]
  node_easy_main["RL runner<br/>[main.py]"]
  node_worker_group["Ray worker groups<br/>[worker_group.py]"]
  node_protocol["Training data protocol<br/>[protocol.py]"]
  node_fsdp_workers["FSDP actor workers<br/>[fsdp_workers.py]"]
  node_rollout["vLLM rollout"]
  node_checkpoint["FSDP checkpoints"]
  node_reward["Reward functions<br/>[function.py]"]
end

subgraph group_trl["TRL Library"]
  node_trl_core["RL algorithms<br/>[core.py]"]
  node_ppo["PPO trainer<br/>[ppo_trainer.py]"]
  node_dpo["DPO trainer<br/>[dpo_trainer.py]"]
  node_trl_models["Model wrappers<br/>[modeling_base.py]"]
end

node_user(("User"))
node_image_video(("Image/video input"))
node_critic_data["Critic pairwise data"]

node_user -->|"submits prompt"| node_serve_clients
node_image_video -->|"preprocesses media"| node_mm_utils
node_serve_clients -.->|"requests inference"| node_serve_controller
node_serve_controller -.->|"dispatches requests"| node_serve_worker
node_serve_worker -->|"loads model"| node_model_builder
node_serve_worker -->|"processes inputs"| node_mm_utils
node_model_builder -->|"selects variant"| node_language_models
node_llava_arch -->|"builds encoder"| node_vision
node_llava_arch -->|"builds projector"| node_projector
node_llava_arch -->|"builds resampler"| node_resampler
node_model_builder -->|"loads architecture"| node_llava_arch
node_evaluation -.->|"loads model"| node_model_builder
node_training_entry -->|"configures trainer"| node_llava_trainer
node_critic_data -.->|"trains critic"| node_easy_main
node_easy_main -->|"creates workers"| node_worker_group
node_easy_main -->|"configures roles"| node_fsdp_workers
node_easy_main -->|"creates reward manager"| node_reward
node_fsdp_workers -->|"uses batches"| node_protocol
node_fsdp_workers -->|"runs generation"| node_rollout
node_fsdp_workers -.->|"saves state"| node_checkpoint
node_ppo -->|"uses utilities"| node_trl_core
node_ppo -->|"uses wrappers"| node_trl_models
node_dpo -.->|"uses algorithms"| node_trl_core
node_training_entry -.->|"uses training utilities"| node_dpo

click node_mm_utils "https://github.com/llava-vl/llava-next/blob/main/llava/mm_utils.py"
click node_vision "https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_encoder"
click node_projector "https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_projector"
click node_resampler "https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_resampler"
click node_llava_arch "https://github.com/llava-vl/llava-next/blob/main/llava/model/llava_arch.py"
click node_model_builder "https://github.com/llava-vl/llava-next/blob/main/llava/model/builder.py"
click node_language_models "https://github.com/llava-vl/llava-next/tree/main/llava/model/language_model"
click node_serve_worker "https://github.com/llava-vl/llava-next/blob/main/llava/serve/model_worker.py"
click node_serve_controller "https://github.com/llava-vl/llava-next/blob/main/llava/serve/controller.py"
click node_serve_clients "https://github.com/llava-vl/llava-next/blob/main/llava/serve/cli.py"
click node_evaluation "https://github.com/llava-vl/llava-next/blob/main/llava/eval/evaluate_interleave.py"
click node_training_entry "https://github.com/llava-vl/llava-next/blob/main/llava/train/train_dpo.py"
click node_llava_trainer "https://github.com/llava-vl/llava-next/blob/main/llava/train/llava_trainer.py"
click node_easy_main "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/trainer/main.py"
click node_worker_group "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/single_controller/base/worker_group.py"
click node_protocol "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/protocol.py"
click node_fsdp_workers "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py"
click node_rollout "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/rollout/vllm_rollout_spmd.py"
click node_checkpoint "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/utils/checkpoint/fsdp_checkpoint_manager.py"
click node_reward "https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/reward/function.py"
click node_trl_core "https://github.com/llava-vl/llava-next/blob/main/trl/core.py"
click node_ppo "https://github.com/llava-vl/llava-next/blob/main/trl/trainer/ppo_trainer.py"
click node_dpo "https://github.com/llava-vl/llava-next/blob/main/trl/trainer/dpo_trainer.py"
click node_trl_models "https://github.com/llava-vl/llava-next/blob/main/trl/models/modeling_base.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mm_utils,node_vision,node_projector,node_resampler,node_llava_arch,node_model_builder,node_language_models,node_user toneBlue
class node_serve_worker,node_serve_controller,node_serve_clients,node_evaluation toneAmber
class node_training_entry,node_llava_trainer toneMint
class node_easy_main,node_worker_group,node_protocol,node_fsdp_workers,node_rollout,node_checkpoint,node_reward toneRose
class node_trl_core,node_ppo,node_dpo,node_trl_models,node_image_video,node_critic_data toneIndigo
```

## Components

### Multimodal Model

- [Media preprocessing](https://github.com/llava-vl/llava-next/blob/main/llava/mm_utils.py): `llava/mm_utils.py`
- [Vision encoders](https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_encoder): `llava/model/multimodal_encoder`
- [Multimodal projector](https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_projector): `llava/model/multimodal_projector`
- [Visual resamplers](https://github.com/llava-vl/llava-next/tree/main/llava/model/multimodal_resampler): `llava/model/multimodal_resampler`
- [LLaVA architecture](https://github.com/llava-vl/llava-next/blob/main/llava/model/llava_arch.py): `llava/model/llava_arch.py`
- [Model loader](https://github.com/llava-vl/llava-next/blob/main/llava/model/builder.py): `llava/model/builder.py`
- [Language model variants](https://github.com/llava-vl/llava-next/tree/main/llava/model/language_model): `llava/model/language_model`

### Inference and Evaluation

- [Model worker](https://github.com/llava-vl/llava-next/blob/main/llava/serve/model_worker.py): `llava/serve/model_worker.py`
- [Serving controller](https://github.com/llava-vl/llava-next/blob/main/llava/serve/controller.py): `llava/serve/controller.py`
- [Chat clients](https://github.com/llava-vl/llava-next/blob/main/llava/serve/cli.py): `llava/serve/cli.py`
- [Interleave evaluation](https://github.com/llava-vl/llava-next/blob/main/llava/eval/evaluate_interleave.py): `llava/eval/evaluate_interleave.py`

### Training

- [DPO training](https://github.com/llava-vl/llava-next/blob/main/llava/train/train_dpo.py): `llava/train/train_dpo.py`
- [LLaVA trainer](https://github.com/llava-vl/llava-next/blob/main/llava/train/llava_trainer.py): `llava/train/llava_trainer.py`

### Distributed RL

- [RL runner](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/trainer/main.py): `llava-critic-r1/EasyR1/verl/trainer/main.py`
- [Ray worker groups](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/single_controller/base/worker_group.py): `llava-critic-r1/EasyR1/verl/single_controller/base/worker_group.py`
- [Training data protocol](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/protocol.py): `llava-critic-r1/EasyR1/verl/protocol.py`
- [FSDP actor workers](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py): `llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py`
- [vLLM rollout](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/rollout/vllm_rollout_spmd.py): `llava-critic-r1/EasyR1/verl/workers/rollout/vllm_rollout_spmd.py`
- [FSDP checkpoints](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/utils/checkpoint/fsdp_checkpoint_manager.py): `llava-critic-r1/EasyR1/verl/utils/checkpoint/fsdp_checkpoint_manager.py`
- [Reward functions](https://github.com/llava-vl/llava-next/blob/main/llava-critic-r1/EasyR1/verl/workers/reward/function.py): `llava-critic-r1/EasyR1/verl/workers/reward/function.py`

### TRL Library

- [RL algorithms](https://github.com/llava-vl/llava-next/blob/main/trl/core.py): `trl/core.py`
- [PPO trainer](https://github.com/llava-vl/llava-next/blob/main/trl/trainer/ppo_trainer.py): `trl/trainer/ppo_trainer.py`
- [DPO trainer](https://github.com/llava-vl/llava-next/blob/main/trl/trainer/dpo_trainer.py): `trl/trainer/dpo_trainer.py`
- [Model wrappers](https://github.com/llava-vl/llava-next/blob/main/trl/models/modeling_base.py): `trl/models/modeling_base.py`

### Other components

- **User**
- **Image/video input**
- **Critic pairwise data**

## Connections

- User → Chat clients: submits prompt
- Image/video input → Media preprocessing: preprocesses media (evidence: `llava/mm_utils.py`)
- Chat clients → Serving controller: requests inference
- Serving controller → Model worker: dispatches requests
- Model worker → Model loader: loads model (evidence: `llava/serve/model_worker.py`)
- Model worker → Media preprocessing: processes inputs (evidence: `llava/serve/model_worker.py`)
- Model loader → Language model variants: selects variant (evidence: `llava/model/builder.py`)
- LLaVA architecture → Vision encoders: builds encoder (evidence: `llava/model/llava_arch.py`)
- LLaVA architecture → Multimodal projector: builds projector (evidence: `llava/model/llava_arch.py`)
- LLaVA architecture → Visual resamplers: builds resampler (evidence: `llava/model/llava_arch.py`)
- Model loader → LLaVA architecture: loads architecture (evidence: `llava/model/builder.py`)
- Interleave evaluation → Model loader: loads model
- DPO training → LLaVA trainer: configures trainer (evidence: `llava/train/train_dpo.py`)
- Critic pairwise data → RL runner: trains critic
- RL runner → Ray worker groups: creates workers (evidence: `llava-critic-r1/EasyR1/verl/trainer/main.py`)
- RL runner → FSDP actor workers: configures roles (evidence: `llava-critic-r1/EasyR1/verl/trainer/main.py`)
- RL runner → Reward functions: creates reward manager (evidence: `llava-critic-r1/EasyR1/verl/trainer/main.py`)
- FSDP actor workers → Training data protocol: uses batches (evidence: `llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py`)
- FSDP actor workers → vLLM rollout: runs generation (evidence: `llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py`)
- FSDP actor workers → FSDP checkpoints: saves state (evidence: `llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py`)
- PPO trainer → RL algorithms: uses utilities (evidence: `trl/trainer/ppo_trainer.py`)
- PPO trainer → Model wrappers: uses wrappers (evidence: `trl/trainer/ppo_trainer.py`)
- DPO trainer → RL algorithms: uses algorithms
- DPO training → DPO trainer: uses training utilities
