How neongeckocom/neon-llm-vllm works
Project type and purpose
• This is a containerized Python microservice (“neon_llm_vllm”) that proxies incoming LLM query requests—delivered over RabbitMQ—to an external VLLM HTTP server and returns LLM responses back via RabbitMQ. It’s designed for use in Docker environments and integrates with NeonAI’s broader MQ-based infrastructure.
File structure
• Top-level artifacts:
• Dockerfile, .dockerignore, docker_overlay/: Docker build & overlay for configuration injection
• requirements/: pinned Python dependencies
• setup.py, version.py: package metadata & distribution
• neon_llm_vllm/: Python package containing the core logic
• .github/workflows/: CI for tests, releases, Docker image publishing
• Inside neon_llm_vllm/:
• __main__.py: entrypoint that boots the service
• rmq.py: RabbitMQ connection, queue consumers/producers
• vllm.py: HTTP client that forwards queries to the VLLM server and formats responses
README for additional insights
• Protocol: JSON messages with “history” and “query” fields
• Configurable via YAML (neon/diana.yaml) mounted into /config
• Requires MQ host/port & credentials, and VLLM server URL + LLM parameters
Main components & relationships
a. Components:
1. RabbitMQ (external message broker)
2. neon_llm_vllm Container
– RMQ Consumer (rmq.py)
– VLLM Client (vllm.py)
– Configuration (mounted /config/neon/diana.yaml)
3. External VLLM HTTP Server
4. (Optional) Chatbot persona configs under llm_bots
b. Data flow:
1. Client publishes JSON request → RabbitMQ “vllm” queue
2. RMQ Consumer in neon_llm_vllm dequeues request
3. VLLM Client calls HTTP POST → VLLM Server (/generate)
4. VLLM Server returns LLM output
5. RMQ Producer publishes response back to a reply-to queue or same exchange
c. Architectural patterns & principles:
– Microservice: single responsibility proxy
– Message-driven: decouples clients from LLM server
– Twelve-factor config: YAML file mounted at runtime
– Containerized deployment with overlay filesystem
d. Key technologies:
– Python, pika (RabbitMQ client)
– HTTP for VLLM server calls
– Docker + overlay config
– GitHub Actions for CI/CD
Detailed callouts
• How docker_overlay injects /config/neon/diana.yaml before container start