# neongeckocom/neon-llm-vllm architecture

> Architecture diagram of the GitHub repository neongeckocom/neon-llm-vllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neongeckocom/neon-llm-vllm
- Repository: https://github.com/neongeckocom/neon-llm-vllm

## Overview

##### Project type and purpose
- This is a containerized Python microservice (“neon_llm_vllm”) that proxies incoming LLM query requests—delivered over RabbitMQ—to an external VLLM HTTP server and returns LLM responses back via RabbitMQ. It’s designed for use in Docker environments and integrates with NeonAI’s broader MQ-based infrastructure.

##### File structure
- Top-level artifacts:
- Dockerfile, .dockerignore, docker_overlay/: Docker build & overlay for configuration injection
- requirements/: pinned Python dependencies
- setup.py, version.py: package metadata & distribution
- neon_llm_vllm/: Python package containing the core logic
- .github/workflows/: CI for tests, releases, Docker image publishing
- Inside neon_llm_vllm/:
- __main__.py: entrypoint that boots the service
- rmq.py: RabbitMQ connection, queue consumers/producers
- vllm.py: HTTP client that forwards queries to the VLLM server and formats responses

##### README for additional insights
- Protocol: JSON messages with “history” and “query” fields
- Configurable via YAML (neon/diana.yaml) mounted into /config
- Requires MQ host/port & credentials, and VLLM server URL + LLM parameters

##### Main components & relationships
a. Components:
1. RabbitMQ (external message broker)
2. neon_llm_vllm Container
­– RMQ Consumer (rmq.py)
­– VLLM Client (vllm.py)
­– Configuration (mounted /config/neon/diana.yaml)
3. External VLLM HTTP Server
4. (Optional) Chatbot persona configs under llm_bots
b. Data flow:
1. Client publishes JSON request → RabbitMQ “vllm” queue
2. RMQ Consumer in neon_llm_vllm dequeues request
3. VLLM Client calls HTTP POST → VLLM Server (/generate)
4. VLLM Server returns LLM output
5. RMQ Producer publishes response back to a reply-to queue or same exchange
c. Architectural patterns & principles:
– Microservice: single responsibility proxy
– Message-driven: decouples clients from LLM server
– Twelve-factor config: YAML file mounted at runtime
– Containerized deployment with overlay filesystem
d. Key technologies:
– Python, pika (RabbitMQ client)
– HTTP for VLLM server calls
– Docker + overlay config
– GitHub Actions for CI/CD

##### Detailed callouts
- How docker_overlay injects /config/neon/diana.yaml before container start

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Infrastructure
    RabbitMQ["RabbitMQ\n(Message Broker)"]:::infra
    VLLMServer["VLLM HTTP Server\n(/generate)"]:::infra

    %% Configuration Injection
    docker_overlay["docker_overlay/opt/neon/diana.yaml"]:::config
    Config["Config YAML\n(/config/neon/diana.yaml)"]:::config

    %% Containerized Service
    subgraph "neon_llm_vllm Container" 
        direction TB
        main["__main__.py\n(Service Entrypoint)"]:::code
        rmq["rmq.py\n(RMQ Consumer/Producer)"]:::code
        vllm["vllm.py\n(VLLM HTTP Client)"]:::code
    end

    %% Repository Artifacts
    subgraph "Repository Artifacts"
        direction TB
        setup["setup.py"]:::code
        version["version.py"]:::code
        requirements["requirements/requirements.txt"]:::code
        workflows[".github/workflows/"]:::code
    end

    %% Data Flows
    RabbitMQ -->|"Consume JSON request"| rmq
    rmq -->|"POST /generate"| VLLMServer
    VLLMServer -->|"LLM response"| rmq
    rmq -->|"Publish JSON response"| RabbitMQ

    %% Config & Build
    docker_overlay --> Config
    Config -->|"Mounted at /config"| main
    Dockerfile["Dockerfile"]:::infra -->|"Build Image"| main

    %% Click Events
    click Dockerfile "https://github.com/neongeckocom/neon-llm-vllm/tree/dev/Dockerfile"
    click docker_overlay "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/docker_overlay/opt/neon/diana.yaml"
    click rmq "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/neon_llm_vllm/rmq.py"
    click vllm "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/neon_llm_vllm/vllm.py"
    click main "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/neon_llm_vllm/__main__.py"
    click setup "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/setup.py"
    click version "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/version.py"
    click requirements "https://github.com/neongeckocom/neon-llm-vllm/blob/dev/requirements/requirements.txt"
    click workflows "https://github.com/neongeckocom/neon-llm-vllm/tree/dev/.github/workflows/"

    %% Styles
    classDef infra fill:#fdebd0,stroke:#d35400,color:#000;
    classDef service fill:#d6eaf8,stroke:#21618c,color:#000;
    classDef code fill:#d5f5e3,stroke:#27ae60,color:#000;
    classDef config fill:#fcf3cf,stroke:#b7950b,color:#000;
```
