# centralesupelec/aristote-dispatcher architecture

> Architecture diagram of the GitHub repository centralesupelec/aristote-dispatcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/centralesupelec/aristote-dispatcher
- Repository: https://github.com/centralesupelec/aristote-dispatcher

## Overview

##### Project type and purpose
- This is a Kubernetes-centric, full-stack orchestration system for large-scale LLM inference.
- It uses a Helm chart to deploy:
- A Python “sender” microservice exposing an OpenAI-compatible API and queuing logic
- A RabbitMQ cluster (internal or external) for prioritized task queues
- Multiple Python “consumer” workers that manage LLM load and token-based priorities
- A vLLM inference server using GPUs
- A relational database (MySQL/PostgreSQL) for authentication tokens
- Optional monitoring stack (Prometheus + Grafana) and ingress (nginx)

##### File structure
- Top-level config:
- `.github/workflows/ci.yml`, `mkdocs.yml`, `pyproject.toml`, `pre-commit` – CI, docs, build setup
- `docs/` – architecture sketch, troubleshooting, mkdocs site
- `grafana/` – dashboards & alert groups JSON
- `workers/compose.yml` orchestrates local dev Docker setup
- `workers/sender/` – Python Flask/FastAPI service, RPC client, DB models
- `workers/consumer/` – Python service, RPC server, metrics, probes, selection strategies
- `workers/init_db/` – example SQL for DB initialization
- Helm‐specific configuration lives in root `values` blocks described in README

##### README for additional insights
- The README outlines the 8-step request flow, from user → API → RabbitMQ → consumer → back to sender → vLLM → user
- It lists environment variables for each component, Helm customization options, and prerequisites (GPU operator, RabbitMQ operator, ingress-nginx, Prometheus operator)

a. Main components
1. Client (OpenAI-compliant user)
2. Ingress (nginx)
3. Sender API service
4. RabbitMQ cluster
5. Consumer workers
6. vLLM inference server(s)
7. Database (MySQL/Postgres)
8. Monitoring (Prometheus & Grafana)
b. Relationships & data flows
1. User → Ingress → Sender (HTTP OpenAI API)
2. Sender → DB (validate token & fetch priority)
3. Sender → RabbitMQ (RPC publish with priority)
4. RabbitMQ → Consumer (dequeue when capacity permits)
5. Consumer → Sender (RPC callback with LLM address)
6. Sender → vLLM inference server (proxy the request)
7. vLLM → Sender → User (HTTP response)
8. Prometheus scrapes metrics endpoints on RabbitMQ, sender, consumer, vLLM
9. Grafana queries Prometheus for dashboards/alerts
c. Architectural patterns & principles
- Producer–consumer via RabbitMQ with per-model queues
- RPC callbacks for dynamic load-aware allocation
- Token-based priority & threshold enforcement at the edge
- Separation of concerns: authentication & queuing (sender) vs. throughput management (consumer) vs. inference (vLLM)
- Helm chart organizes all k8s resources; operators manage RabbitMQ, GPUs, Prometheus
d. Key technologies & frameworks
- Python microservices (FastAPI or Flask)
- RabbitMQ for priority queueing & RPC
- vLLM inference server (OpenAI-compatible)
- MySQL/PostgreSQL for token storage
- Kubernetes + Helm + operators (GPU, RabbitMQ, Prometheus)
- Ingress-nginx, Prometheus, Grafana

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client and Ingress
    Client["Client (OpenAI User)"]:::external
    Ingress["Ingress (nginx)"]:::infra
    Client -->|"HTTP POST /v1/chat/completions"| Ingress

    %% Kubernetes Namespace
    subgraph "K8s Namespace" 
        direction TB

        %% Sender Subchart
        subgraph "Helm: sender" 
            direction TB
            SenderDep["Deployment: Sender API"]:::api
            SenderSvc["Service: sender-svc"]:::api
            SenderCM["ConfigMap/Secrets"]:::config
            SenderDep --> SenderSvc
            SenderDep --> SenderCM
        end

        %% Database
        TokenDB["Token DB (MySQL/Postgres)"]:::data
        SenderSvc -->|"Validate token & fetch priority (SQL)"| TokenDB

        %% RabbitMQ Cluster
        RabbitMQ["RabbitMQ Cluster"]:::messaging
        SenderSvc -->|"AMQP publish priority"| RabbitMQ

        %% Consumer Subchart
        subgraph "Helm: consumer" 
            direction TB
            ConsumerDep["Deployment: Consumer Worker"]:::worker
            ConsumerSvc["Service: consumer-svc"]:::worker
            ConsumerDep --> ConsumerSvc
        end

        RabbitMQ -->|"AMQP consume"| ConsumerDep
        ConsumerDep -->|"RPC callback"| SenderSvc

        %% vLLM Inference Subchart
        subgraph "Helm: inferenceserver" 
            direction TB
            vLLMDep["Deployment: vLLM Server"]:::inference
            vLLMSvc["Service: vllm-svc"]:::inference
            vLLMDep --> vLLMSvc
        end

        SenderSvc -->|"HTTP proxy"| vLLMSvc
        vLLMSvc -->|"HTTP response"| SenderSvc
    end

    %% Monitoring
    subgraph Monitoring
        direction TB
        Prometheus["Prometheus"]:::monitor
        Grafana["Grafana"]:::monitor
    end
    Prometheus -.->|scrape metrics| RabbitMQ
    Prometheus -.->|scrape metrics| SenderSvc
    Prometheus -.->|scrape metrics| ConsumerSvc
    Prometheus -.->|scrape metrics| vLLMSvc
    Grafana -->|queries| Prometheus

    %% Click Events
    click SenderDep "https://github.com/centralesupelec/aristote-dispatcher/blob/main/workers/sender/main.py"
    click ConsumerDep "https://github.com/centralesupelec/aristote-dispatcher/blob/main/workers/consumer/main.py"
    click RabbitMQ "https://github.com/centralesupelec/aristote-dispatcher/blob/main/workers/compose.yml"
    click vLLMDep "https://github.com/centralesupelec/aristote-dispatcher/blob/main/workers/consumer/vllm_server.py"
    click TokenDB "https://github.com/centralesupelec/aristote-dispatcher/blob/main/workers/init_db/init_mysql.sql.example"

    %% Styles
    classDef api fill:#D0E6FF,stroke:#5691F9,stroke-width:1px
    classDef worker fill:#E6F7FF,stroke:#4A90E2,stroke-width:1px
    classDef messaging fill:#FFE5CC,stroke:#F2994A,stroke-width:1px
    classDef inference fill:#F3E5FF,stroke:#9B59B6,stroke-width:1px
    classDef data fill:#E5FFCC,stroke:#27AE60,stroke-width:1px
    classDef monitor fill:#F2F2F2,stroke:#7F8C8D,stroke-width:1px,stroke-dasharray: 5 5
    classDef external fill:#FFFFFF,stroke:#000000,stroke-width:1px,stroke-dasharray: 2 2
    classDef infra fill:#F0F0F0,stroke:#BDC3C7,stroke-width:1px
    classDef config fill:#FFF5CC,stroke:#F1C40F,stroke-width:1px
```
