How mmgxa/llava_vllm works
Project type and purpose
• This repo is a demonstration/deployment wrapper for serving a LLaVA large-language-vision model on top of the vLLM inference engine.
• It includes the full vLLM library (Python + custom CUDA kernels), two HTTP server entrypoints (a LLaVA-specific server and an OpenAI-compatible server), a Streamlit UI, and test clients.
File structure
Top level
• main.py – Streamlit frontend
• test_server.py, test_openai.py – integration tests against your servers
• vllm/ – vendored vLLM package with all source code, CUDA kernels, docs, examples, entrypoints, tests, and build configs
Inside vllm/entrypoints
• llava_server.py – launches HTTP server for LLaVA inference
• openai/lapi_servers.py – drop-in OpenAI API server
Inside vllm/vllm/
• engine/ (async & sync engines, scheduler, metrics)
• model_executor/ (model loader, layer implementations, sampling logic)
• worker/ (processes/threads for model execution)
Inside vllm/csrc/
• CUDA kernels and pybind bindings for attention, layernorm, quantization, etc.
README for additional insights
• You run servers via Python CLI modules under vllm.entrypoints, pointing at a HF-hub model.