How haotian-liu/llava works
LLaVA is a multimodal assistant that combines image processing with language-model inference, and also provides model training and benchmark evaluation. A typical user flow is to submit an image and prompt, load a pretrained model, encode image and text inputs, and generate a response; hosted serving adds a web UI, controller, and model workers. The README documents training and evaluation capabilities, represented here at subsystem level. Several evaluation and training details are unsampled, so their internal links are omitted rather than inferred.
Main entry points: llava_arch.py Multimodal Architecture · predict.py Prediction API · model_worker.py Model Worker · builder.py Model Loader
Components 20
Model Inference
- Prediction API
predict.py - Model Loader
llava/model/builder.py - Conversation Formats
llava/conversation.py - Image and Token Utils
llava/mm_utils.py - Multimodal Architecture
llava/model/llava_arch.py - Vision Encoder Builder
llava/model/multimodal_encoder/builder.py - CLIP Encoder
llava/model/multimodal_encoder/clip_encoder.py - Multimodal Projector
llava/model/multimodal_projector/builder.py - LLaMA Model
llava/model/language_model/llava_llama.py - Mistral Model
llava/model/language_model/llava_mistral.py - MPT Model
llava/model/language_model/llava_mpt.py
Hosted Serving
- Gradio Web UI
llava/serve/gradio_web_server.py - Worker Controller
llava/serve/controller.py - Model Worker
llava/serve/model_worker.py - SGLang Worker
llava/serve/sglang_worker.py
Model Training
- Training Engine
llava/train/llava_trainer.py - Training Pipeline
llava/train/train.py
Evaluation
- Benchmark Evaluation
llava/eval
Data Preparation
- Dataset Conversion
scripts/convert_sqa_to_llava_base_prompt.py
Other components
- User
Connections 21
- User → Prediction API: submits prompt
predict.py - Prediction API → Model Loader: loads model
predict.py - Prediction API → Conversation Formats: formats prompt
predict.py - Prediction API → Image and Token Utils: tokenizes image
predict.py - Model Loader → LLaMA Model: loads model
llava/model/builder.py - LLaMA Model → Multimodal Architecture: uses architecture
llava/model/language_model/llava_llama.py - Mistral Model → Multimodal Architecture: uses architecture
llava/model/language_model/llava_mistral.py - MPT Model → Multimodal Architecture: uses architecture
llava/model/language_model/llava_mpt.py - Multimodal Architecture → Vision Encoder Builder: builds encoder
llava/model/llava_arch.py - Multimodal Architecture → Multimodal Projector: builds projector
llava/model/llava_arch.py - Vision Encoder Builder → CLIP Encoder: selects encoder
llava/model/multimodal_encoder/builder.py - Model Worker → Model Loader: loads model
llava/serve/model_worker.py - Model Worker → Image and Token Utils: processes image
llava/serve/model_worker.py - Worker Controller → Model Worker: forwards generation
llava/serve/controller.py - Worker Controller → Model Worker: checks status
llava/serve/controller.py - Training Pipeline → Training Engine: uses trainer
llava/train/train.py - Training Pipeline → Conversation Formats: formats examples
llava/train/train.py - Training Pipeline → Image and Token Utils: processes images
llava/train/train.py - Benchmark Evaluation → Model Loader: loads model
llava/eval/model_vqa_mmbench.py - Benchmark Evaluation → Conversation Formats: formats prompts
llava/eval/model_vqa_mmbench.py - Benchmark Evaluation → Image and Token Utils: processes images
llava/eval/model_vqa_mmbench.py