# haotian-liu/llava architecture

> Architecture diagram of the GitHub repository haotian-liu/llava, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haotian-liu/llava
- Repository: https://github.com/haotian-liu/llava

## Overview

LLaVA is a multimodal assistant that combines image processing with language-model inference, and also provides model training and benchmark evaluation. A typical user flow is to submit an image and prompt, load a pretrained model, encode image and text inputs, and generate a response; hosted serving adds a web UI, controller, and model workers. The README documents training and evaluation capabilities, represented here at subsystem level. Several evaluation and training details are unsampled, so their internal links are omitted rather than inferred.

Main entry points: [`llava/model/llava_arch.py`](https://github.com/haotian-liu/llava/blob/main/llava/model/llava_arch.py) (Multimodal Architecture), [`predict.py`](https://github.com/haotian-liu/llava/blob/main/predict.py) (Prediction API), [`llava/serve/model_worker.py`](https://github.com/haotian-liu/llava/blob/main/llava/serve/model_worker.py) (Model Worker), [`llava/model/builder.py`](https://github.com/haotian-liu/llava/blob/main/llava/model/builder.py) (Model Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Model Inference"]
  node_predict["Prediction API<br/>[predict.py]"]
  node_model_loader["Model Loader<br/>[builder.py]"]
  node_conversation["Conversation Formats<br/>[conversation.py]"]
  node_image_utils["Image and Token Utils<br/>[mm_utils.py]"]
  node_llava_arch["Multimodal Architecture<br/>[llava_arch.py]"]
  node_vision_builder["Vision Encoder Builder<br/>[builder.py]"]
  node_clip_encoder["CLIP Encoder<br/>[clip_encoder.py]"]
  node_projector["Multimodal Projector<br/>[builder.py]"]
  node_llama_model["LLaMA Model<br/>[llava_llama.py]"]
  node_mistral_model["Mistral Model<br/>[llava_mistral.py]"]
  node_mpt_model["MPT Model<br/>[llava_mpt.py]"]
end

subgraph group_serving["Hosted Serving"]
  node_web_ui["Gradio Web UI"]
  node_controller["Worker Controller<br/>[controller.py]"]
  node_model_worker["Model Worker<br/>[model_worker.py]"]
  node_sglang_worker["SGLang Worker<br/>[sglang_worker.py]"]
end

subgraph group_training["Model Training"]
  node_trainer["Training Engine<br/>[llava_trainer.py]"]
  node_train_pipeline["Training Pipeline<br/>[train.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_suite["Benchmark Evaluation"]
end

subgraph group_data["Data Preparation"]
  node_conversion["Dataset Conversion"]
end

node_user(("User"))

node_user -->|"submits prompt"| node_predict
node_predict -->|"loads model"| node_model_loader
node_predict -->|"formats prompt"| node_conversation
node_predict -->|"tokenizes image"| node_image_utils
node_model_loader -->|"loads model"| node_llama_model
node_llama_model -->|"uses architecture"| node_llava_arch
node_mistral_model -->|"uses architecture"| node_llava_arch
node_mpt_model -->|"uses architecture"| node_llava_arch
node_llava_arch -->|"builds encoder"| node_vision_builder
node_llava_arch -->|"builds projector"| node_projector
node_vision_builder -->|"selects encoder"| node_clip_encoder
node_model_worker -->|"loads model"| node_model_loader
node_model_worker -->|"processes image"| node_image_utils
node_controller -->|"forwards generation"| node_model_worker
node_controller -->|"checks status"| node_model_worker
node_train_pipeline -->|"uses trainer"| node_trainer
node_train_pipeline -->|"formats examples"| node_conversation
node_train_pipeline -->|"processes images"| node_image_utils
node_eval_suite -->|"loads model"| node_model_loader
node_eval_suite -->|"formats prompts"| node_conversation
node_eval_suite -->|"processes images"| node_image_utils

click node_predict "https://github.com/haotian-liu/llava/blob/main/predict.py"
click node_model_loader "https://github.com/haotian-liu/llava/blob/main/llava/model/builder.py"
click node_conversation "https://github.com/haotian-liu/llava/blob/main/llava/conversation.py"
click node_image_utils "https://github.com/haotian-liu/llava/blob/main/llava/mm_utils.py"
click node_llava_arch "https://github.com/haotian-liu/llava/blob/main/llava/model/llava_arch.py"
click node_vision_builder "https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_encoder/builder.py"
click node_clip_encoder "https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_encoder/clip_encoder.py"
click node_projector "https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_projector/builder.py"
click node_llama_model "https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_llama.py"
click node_mistral_model "https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_mistral.py"
click node_mpt_model "https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_mpt.py"
click node_web_ui "https://github.com/haotian-liu/llava/blob/main/llava/serve/gradio_web_server.py"
click node_controller "https://github.com/haotian-liu/llava/blob/main/llava/serve/controller.py"
click node_model_worker "https://github.com/haotian-liu/llava/blob/main/llava/serve/model_worker.py"
click node_sglang_worker "https://github.com/haotian-liu/llava/blob/main/llava/serve/sglang_worker.py"
click node_trainer "https://github.com/haotian-liu/llava/blob/main/llava/train/llava_trainer.py"
click node_train_pipeline "https://github.com/haotian-liu/llava/blob/main/llava/train/train.py"
click node_eval_suite "https://github.com/haotian-liu/llava/tree/main/llava/eval"
click node_conversion "https://github.com/haotian-liu/llava/blob/main/scripts/convert_sqa_to_llava_base_prompt.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_predict,node_model_loader,node_conversation,node_image_utils,node_llava_arch,node_vision_builder,node_clip_encoder,node_projector,node_llama_model,node_mistral_model,node_mpt_model,node_user toneBlue
class node_web_ui,node_controller,node_model_worker,node_sglang_worker toneAmber
class node_trainer,node_train_pipeline toneMint
class node_eval_suite toneRose
class node_conversion toneIndigo
```

## Components

### Model Inference

- [Prediction API](https://github.com/haotian-liu/llava/blob/main/predict.py): `predict.py`
- [Model Loader](https://github.com/haotian-liu/llava/blob/main/llava/model/builder.py): `llava/model/builder.py`
- [Conversation Formats](https://github.com/haotian-liu/llava/blob/main/llava/conversation.py): `llava/conversation.py`
- [Image and Token Utils](https://github.com/haotian-liu/llava/blob/main/llava/mm_utils.py): `llava/mm_utils.py`
- [Multimodal Architecture](https://github.com/haotian-liu/llava/blob/main/llava/model/llava_arch.py): `llava/model/llava_arch.py`
- [Vision Encoder Builder](https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_encoder/builder.py): `llava/model/multimodal_encoder/builder.py`
- [CLIP Encoder](https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_encoder/clip_encoder.py): `llava/model/multimodal_encoder/clip_encoder.py`
- [Multimodal Projector](https://github.com/haotian-liu/llava/blob/main/llava/model/multimodal_projector/builder.py): `llava/model/multimodal_projector/builder.py`
- [LLaMA Model](https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_llama.py): `llava/model/language_model/llava_llama.py`
- [Mistral Model](https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_mistral.py): `llava/model/language_model/llava_mistral.py`
- [MPT Model](https://github.com/haotian-liu/llava/blob/main/llava/model/language_model/llava_mpt.py): `llava/model/language_model/llava_mpt.py`

### Hosted Serving

- [Gradio Web UI](https://github.com/haotian-liu/llava/blob/main/llava/serve/gradio_web_server.py): `llava/serve/gradio_web_server.py`
- [Worker Controller](https://github.com/haotian-liu/llava/blob/main/llava/serve/controller.py): `llava/serve/controller.py`
- [Model Worker](https://github.com/haotian-liu/llava/blob/main/llava/serve/model_worker.py): `llava/serve/model_worker.py`
- [SGLang Worker](https://github.com/haotian-liu/llava/blob/main/llava/serve/sglang_worker.py): `llava/serve/sglang_worker.py`

### Model Training

- [Training Engine](https://github.com/haotian-liu/llava/blob/main/llava/train/llava_trainer.py): `llava/train/llava_trainer.py`
- [Training Pipeline](https://github.com/haotian-liu/llava/blob/main/llava/train/train.py): `llava/train/train.py`

### Evaluation

- [Benchmark Evaluation](https://github.com/haotian-liu/llava/tree/main/llava/eval): `llava/eval`

### Data Preparation

- [Dataset Conversion](https://github.com/haotian-liu/llava/blob/main/scripts/convert_sqa_to_llava_base_prompt.py): `scripts/convert_sqa_to_llava_base_prompt.py`

### Other components

- **User**

## Connections

- User → Prediction API: submits prompt (evidence: `predict.py`)
- Prediction API → Model Loader: loads model (evidence: `predict.py`)
- Prediction API → Conversation Formats: formats prompt (evidence: `predict.py`)
- Prediction API → Image and Token Utils: tokenizes image (evidence: `predict.py`)
- Model Loader → LLaMA Model: loads model (evidence: `llava/model/builder.py`)
- LLaMA Model → Multimodal Architecture: uses architecture (evidence: `llava/model/language_model/llava_llama.py`)
- Mistral Model → Multimodal Architecture: uses architecture (evidence: `llava/model/language_model/llava_mistral.py`)
- MPT Model → Multimodal Architecture: uses architecture (evidence: `llava/model/language_model/llava_mpt.py`)
- Multimodal Architecture → Vision Encoder Builder: builds encoder (evidence: `llava/model/llava_arch.py`)
- Multimodal Architecture → Multimodal Projector: builds projector (evidence: `llava/model/llava_arch.py`)
- Vision Encoder Builder → CLIP Encoder: selects encoder (evidence: `llava/model/multimodal_encoder/builder.py`)
- Model Worker → Model Loader: loads model (evidence: `llava/serve/model_worker.py`)
- Model Worker → Image and Token Utils: processes image (evidence: `llava/serve/model_worker.py`)
- Worker Controller → Model Worker: forwards generation (evidence: `llava/serve/controller.py`)
- Worker Controller → Model Worker: checks status (evidence: `llava/serve/controller.py`)
- Training Pipeline → Training Engine: uses trainer (evidence: `llava/train/train.py`)
- Training Pipeline → Conversation Formats: formats examples (evidence: `llava/train/train.py`)
- Training Pipeline → Image and Token Utils: processes images (evidence: `llava/train/train.py`)
- Benchmark Evaluation → Model Loader: loads model (evidence: `llava/eval/model_vqa_mmbench.py`)
- Benchmark Evaluation → Conversation Formats: formats prompts (evidence: `llava/eval/model_vqa_mmbench.py`)
- Benchmark Evaluation → Image and Token Utils: processes images (evidence: `llava/eval/model_vqa_mmbench.py`)
