# mmgxa/llava_vllm architecture

> Architecture diagram of the GitHub repository mmgxa/llava_vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mmgxa/llava_vllm
- Repository: https://github.com/mmgxa/llava_vllm

## Overview

##### Project type and purpose
- This repo is a demonstration/deployment wrapper for serving a LLaVA large-language-vision model on top of the vLLM inference engine.
- It includes the full vLLM library (Python + custom CUDA kernels), two HTTP server entrypoints (a LLaVA-specific server and an OpenAI-compatible server), a Streamlit UI, and test clients.

##### File structure
Top level
- main.py – Streamlit frontend
- test_server.py, test_openai.py – integration tests against your servers
- vllm/ – vendored vLLM package with all source code, CUDA kernels, docs, examples, entrypoints, tests, and build configs
Inside vllm/entrypoints
- llava_server.py – launches HTTP server for LLaVA inference
- openai/lapi_servers.py – drop-in OpenAI API server
Inside vllm/vllm/
- engine/ (async & sync engines, scheduler, metrics)
- model_executor/ (model loader, layer implementations, sampling logic)
- worker/ (processes/threads for model execution)
Inside vllm/csrc/
- CUDA kernels and pybind bindings for attention, layernorm, quantization, etc.

##### README for additional insights
- You run servers via Python CLI modules under vllm.entrypoints, pointing at a HF-hub model.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "UI Layer"
        UI["Streamlit UI"]:::ui
    end

    %% API Layer
    subgraph "API Layer"
        LLAVA_SRV["LLaVA HTTP Server"]:::api
        OPENAI_SRV["OpenAI-compatible HTTP Server"]:::api
    end

    %% Engine Core
    subgraph "Engine Core"
        SCHED["Scheduler & Request Tracker"]:::engine
        ALSYNC["Async LLM Engine"]:::engine
        SYNC["Sync LLM Engine"]:::engine
        ALLAVAASYNC["Async LLaVA Engine"]:::engine
        LLSYNC["Sync LLaVA Engine"]:::engine
        ML["Model Loader"]:::engine
        LAYERS["Layer Implementations"]:::engine
        SAMPLER["Sampler Logic"]:::engine
    end

    %% Native Kernel & Hardware
    subgraph "Native Kernel & Hardware"
        KERNEL["C++/CUDA Kernel Layer"]:::kernel
        GPU["GPU Hardware"]:::hardware
    end

    %% External Clients & Sources
    subgraph "External Clients & Sources"
        TO["test_openai.py"]:::clients
        TS["test_server.py"]:::clients
        AC["api_client.py"]:::clients
        OC["openai_completion_client.py"]:::clients
        HF["Hugging Face Hub/Cache"]:::external
    end

    %% Flows
    UI -->|HTTP| LLAVA_SRV
    UI -->|HTTP| OPENAI_SRV

    LLAVA_SRV -->|"invoke API"| SCHED
    OPENAI_SRV -->|"invoke API"| SCHED

    SCHED --> ALSYNC
    SCHED --> SYNC
    SCHED --> ALLAVAASYNC
    SCHED --> LLSYNC

    ALSYNC --> ML
    SYNC --> ML
    ALLAVAASYNC --> ML
    LLSYNC --> ML

    ML --> LAYERS
    LAYERS --> SAMPLER
    SAMPLER --> KERNEL
    KERNEL --> GPU

    ML -->|"load weights"| HF

    TO --> OPENAI_SRV
    TS --> LLAVA_SRV
    AC --> LLAVA_SRV
    OC --> OPENAI_SRV

    %% Click Events
    click UI "https://github.com/mmgxa/llava_vllm/blob/main/main.py"
    click LLAVA_SRV "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/entrypoints/llava_server.py"
    click OPENAI_SRV "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/entrypoints/openai/lapi_servers.py"
    click SCHED "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/core/scheduler.py"
    click ALSYNC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/engine/async_llm_engine.py"
    click SYNC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/engine/llm_engine.py"
    click ALLAVAASYNC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/engine/async_llava_engine.py"
    click LLSYNC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/engine/llava_engine.py"
    click ML "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/model_executor/model_loader.py"
    click LAYERS "https://github.com/mmgxa/llava_vllm/tree/main/vllm/vllm/model_executor/layers/"
    click SAMPLER "https://github.com/mmgxa/llava_vllm/blob/main/vllm/vllm/model_executor/layers/sampler.py"
    click KERNEL "https://github.com/mmgxa/llava_vllm/tree/main/vllm/csrc/"
    click TO "https://github.com/mmgxa/llava_vllm/blob/main/test_openai.py"
    click TS "https://github.com/mmgxa/llava_vllm/blob/main/test_server.py"
    click AC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/examples/api_client.py"
    click OC "https://github.com/mmgxa/llava_vllm/blob/main/vllm/examples/openai_completion_client.py"

    %% Styles
    classDef ui fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef api fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef engine fill:#ffe0b3,stroke:#333,stroke-width:1px
    classDef kernel fill:#ffcccc,stroke:#333,stroke-width:1px
    classDef hardware fill:#ff9999,stroke:#333,stroke-width:1px
    classDef clients fill:#e5ccff,stroke:#333,stroke-width:1px
    classDef external fill:#f2f2f2,stroke:#333,stroke-width:1px
```
