How llava-vl/llava-next works
LLaVA-NeXT is a multimodal vision-language model project supporting image, video, multi-image/interleaved inference, evaluation, and fine-tuning; the release notes also describe LLaVA-Critic-R1 training with GRPO. The main flow is user input through serving or evaluation, image/video preprocessing, multimodal model inference, and a generated answer. Separate branches represent supervised/DPO training and EasyR1 distributed RL training, plus the bundled TRL training library. Connections reflect README statements and SOURCE INDEX references; unsampled modules are represented only where the tree or documented capabilities supports them.
Main entry points: main.py RL runner · fsdp_workers.py FSDP actor workers · llava_arch.py LLaVA architecture · builder.py Model loader
Components 27
Multimodal Model
- Media preprocessing
llava/mm_utils.py - Vision encoders
llava/model/multimodal_encoder - Multimodal projector
llava/model/multimodal_projector - Visual resamplers
llava/model/multimodal_resampler - LLaVA architecture
llava/model/llava_arch.py - Model loader
llava/model/builder.py - Language model variants
llava/model/language_model
Inference and Evaluation
- Model worker
llava/serve/model_worker.py - Serving controller
llava/serve/controller.py - Chat clients
llava/serve/cli.py - Interleave evaluation
llava/eval/evaluate_interleave.py
Training
- DPO training
llava/train/train_dpo.py - LLaVA trainer
llava/train/llava_trainer.py
Distributed RL
- RL runner
llava-critic-r1/EasyR1/verl/trainer/main.py - Ray worker groups
llava-critic-r1/EasyR1/verl/single_controller/base/worker_group.py - Training data protocol
llava-critic-r1/EasyR1/verl/protocol.py - FSDP actor workers
llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py - vLLM rollout
llava-critic-r1/EasyR1/verl/workers/rollout/vllm_rollout_spmd.py - FSDP checkpoints
llava-critic-r1/EasyR1/verl/utils/checkpoint/fsdp_checkpoint_manager.py - Reward functions
llava-critic-r1/EasyR1/verl/workers/reward/function.py
TRL Library
- RL algorithms
trl/core.py - PPO trainer
trl/trainer/ppo_trainer.py - DPO trainer
trl/trainer/dpo_trainer.py - Model wrappers
trl/models/modeling_base.py
Other components
- User
- Image/video input
- Critic pairwise data
Connections 24
- User → Chat clients: submits prompt
- Image/video input → Media preprocessing: preprocesses media
llava/mm_utils.py - Chat clients → Serving controller: requests inference
- Serving controller → Model worker: dispatches requests
- Model worker → Model loader: loads model
llava/serve/model_worker.py - Model worker → Media preprocessing: processes inputs
llava/serve/model_worker.py - Model loader → Language model variants: selects variant
llava/model/builder.py - LLaVA architecture → Vision encoders: builds encoder
llava/model/llava_arch.py - LLaVA architecture → Multimodal projector: builds projector
llava/model/llava_arch.py - LLaVA architecture → Visual resamplers: builds resampler
llava/model/llava_arch.py - Model loader → LLaVA architecture: loads architecture
llava/model/builder.py - Interleave evaluation → Model loader: loads model
- DPO training → LLaVA trainer: configures trainer
llava/train/train_dpo.py - Critic pairwise data → RL runner: trains critic
- RL runner → Ray worker groups: creates workers
llava-critic-r1/EasyR1/verl/trainer/main.py - RL runner → FSDP actor workers: configures roles
llava-critic-r1/EasyR1/verl/trainer/main.py - RL runner → Reward functions: creates reward manager
llava-critic-r1/EasyR1/verl/trainer/main.py - FSDP actor workers → Training data protocol: uses batches
llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py - FSDP actor workers → vLLM rollout: runs generation
llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py - FSDP actor workers → FSDP checkpoints: saves state
llava-critic-r1/EasyR1/verl/workers/fsdp_workers.py - PPO trainer → RL algorithms: uses utilities
trl/trainer/ppo_trainer.py - PPO trainer → Model wrappers: uses wrappers
trl/trainer/ppo_trainer.py - DPO trainer → RL algorithms: uses algorithms
- DPO training → DPO trainer: uses training utilities