How centralesupelec/aristote-dispatcher works
Project type and purpose
• This is a Kubernetes-centric, full-stack orchestration system for large-scale LLM inference.
• It uses a Helm chart to deploy:
• A Python “sender” microservice exposing an OpenAI-compatible API and queuing logic
• A RabbitMQ cluster (internal or external) for prioritized task queues
• Multiple Python “consumer” workers that manage LLM load and token-based priorities
• A vLLM inference server using GPUs
• A relational database (MySQL/PostgreSQL) for authentication tokens
• Optional monitoring stack (Prometheus + Grafana) and ingress (nginx)
File structure
• Top-level config:
• .github/workflows/ci.yml, mkdocs.yml, pyproject.toml, pre-commit – CI, docs, build setup
• docs/ – architecture sketch, troubleshooting, mkdocs site
• grafana/ – dashboards & alert groups JSON
• workers/compose.yml orchestrates local dev Docker setup
• workers/sender/ – Python Flask/FastAPI service, RPC client, DB models
• workers/consumer/ – Python service, RPC server, metrics, probes, selection strategies
• workers/init_db/ – example SQL for DB initialization
• Helm‐specific configuration lives in root values blocks described in README
README for additional insights
• The README outlines the 8-step request flow, from user → API → RabbitMQ → consumer → back to sender → vLLM → user
• It lists environment variables for each component, Helm customization options, and prerequisites (GPU operator, RabbitMQ operator, ingress-nginx, Prometheus operator)
a. Main components
1. Client (OpenAI-compliant user)
2. Ingress (nginx)
3. Sender API service
4. RabbitMQ cluster
5. Consumer workers
6. vLLM inference server(s)
7. Database (MySQL/Postgres)
8. Monitoring (Prometheus & Grafana)
b. Relationships & data flows
1. User → Ingress → Sender (HTTP OpenAI API)
2. Sender → DB (validate token & fetch priority)
3. Sender → RabbitMQ (RPC publish with priority)
4. RabbitMQ → Consumer (dequeue when capacity permits)
5. Consumer → Sender (RPC callback with LLM address)
6. Sender → vLLM inference server (proxy the request)
7. vLLM → Sender → User (HTTP response)
8. Prometheus scrapes metrics endpoints on RabbitMQ, sender, consumer, vLLM
9. Grafana queries Prometheus for dashboards/alerts
c. Architectural patterns & principles
• Producer–consumer via RabbitMQ with per-model queues
• RPC callbacks for dynamic load-aware allocation
• Token-based priority & threshold enforcement at the edge
• Separation of concerns: authentication & queuing (sender) vs. throughput management (consumer) vs. inference (vLLM)
• Helm chart organizes all k8s resources; operators manage RabbitMQ, GPUs, Prometheus
d. Key technologies & frameworks
• Python microservices (FastAPI or Flask)
• RabbitMQ for priority queueing & RPC
• vLLM inference server (OpenAI-compatible)
• MySQL/PostgreSQL for token storage
• Kubernetes + Helm + operators (GPU, RabbitMQ, Prometheus)
• Ingress-nginx, Prometheus, Grafana