# rares8921/cost-aware-inference-cluster architecture

> Architecture diagram of the GitHub repository rares8921/cost-aware-inference-cluster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rares8921/cost-aware-inference-cluster
- Repository: https://github.com/rares8921/cost-aware-inference-cluster

## Overview

This prototype accepts inference requests through a tenant-aware router, queues them at the scheduler, and has workers poll, batch, and run model inference before reporting completion. The scheduler also tracks worker heartbeats and makes cost-aware scaling decisions; the README says those decisions are logged, not enacted by creating pods or containers. Redis is shared by rate-limit, queue, and worker-registry state. The graph focuses on runtime paths and documented capabilities; unsampled configuration and monitoring/deployment wiring are not expanded into unverified runtime links.

Main entry points: [`services/scheduler/main.py`](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/main.py) (Scheduler API), [`services/worker/main.py`](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/main.py) (Worker service), [`services/scheduler/queue_manager.py`](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/queue_manager.py) (Priority queue), [`services/scheduler/autoscaler.py`](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/autoscaler.py) (Autoscaler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_routing["Request routing"]
  node_router["Router API<br/>[main.py]"]
  node_tenant["Tenant limits<br/>[tenant_manager.py]"]
end

subgraph group_scheduling["Scheduling and scaling"]
  node_scheduler["Scheduler API<br/>[main.py]"]
  node_queue["Priority queue<br/>[queue_manager.py]"]
  node_registry["Worker registry<br/>[worker_registry.py]"]
  node_autoscaler["Autoscaler<br/>[autoscaler.py]"]
  node_cost["Cost optimizer<br/>[cost_optimizer.py]"]
end

subgraph group_execution["Worker inference"]
  node_worker["Worker service<br/>[main.py]"]
  node_batcher["Dynamic batching<br/>[batch_processor.py]"]
  node_model["Model inference<br/>[model_loader.py]"]
end

subgraph group_state["Redis state"]
  node_redis[("Redis")]
end

node_client(("Inference client"))
node_deployment["Container and Kubernetes configuration"]
node_monitoring["Prometheus and Grafana configuration"]

node_client -->|"submits request"| node_router
node_router -->|"checks quota"| node_tenant
node_router -->|"forwards request"| node_scheduler
node_tenant -.->|"reads and updates limits"| node_redis
node_scheduler -->|"enqueues request"| node_queue
node_scheduler -->|"records heartbeats"| node_registry
node_queue -->|"stores queue state"| node_redis
node_registry -->|"stores worker state"| node_redis
node_scheduler -->|"starts decision loop"| node_autoscaler
node_autoscaler -->|"reads queue metrics"| node_queue
node_autoscaler -->|"reads and removes workers"| node_registry
node_autoscaler -->|"checks budget"| node_cost
node_worker -->|"sends heartbeat"| node_scheduler
node_worker -->|"polls for work"| node_scheduler
node_scheduler -->|"dequeues items"| node_queue
node_worker -->|"adds items"| node_batcher
node_batcher -->|"processes text batches"| node_model
node_worker -->|"reports completion"| node_scheduler
node_scheduler -->|"records completion"| node_queue
node_deployment -.->|"configures service"| node_router
node_deployment -.->|"configures service"| node_scheduler
node_deployment -.->|"configures service"| node_worker
node_monitoring -.->|"scrapes and displays metrics"| node_scheduler

click node_router "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/router/main.py"
click node_tenant "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/router/tenant_manager.py"
click node_scheduler "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/main.py"
click node_queue "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/queue_manager.py"
click node_registry "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/worker_registry.py"
click node_autoscaler "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/autoscaler.py"
click node_cost "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/cost_optimizer.py"
click node_worker "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/main.py"
click node_batcher "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/batch_processor.py"
click node_model "https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/model_loader.py"
click node_deployment "https://github.com/rares8921/cost-aware-inference-cluster/tree/main/kubernetes"
click node_monitoring "https://github.com/rares8921/cost-aware-inference-cluster/tree/main/monitoring"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_router,node_tenant,node_client toneBlue
class node_scheduler,node_queue,node_registry,node_autoscaler,node_cost toneAmber
class node_worker,node_batcher,node_model toneMint
class node_redis toneRose
class node_deployment,node_monitoring toneTeal
```

## Components

### Request routing

- [Router API](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/router/main.py): `services/router/main.py`
- [Tenant limits](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/router/tenant_manager.py): `services/router/tenant_manager.py`

### Scheduling and scaling

- [Scheduler API](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/main.py): `services/scheduler/main.py`
- [Priority queue](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/queue_manager.py): `services/scheduler/queue_manager.py`
- [Worker registry](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/worker_registry.py): `services/scheduler/worker_registry.py`
- [Autoscaler](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/autoscaler.py): `services/scheduler/autoscaler.py`
- [Cost optimizer](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/scheduler/cost_optimizer.py): `services/scheduler/cost_optimizer.py`

### Worker inference

- [Worker service](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/main.py): `services/worker/main.py`
- [Dynamic batching](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/batch_processor.py): `services/worker/batch_processor.py`
- [Model inference](https://github.com/rares8921/cost-aware-inference-cluster/blob/main/services/worker/model_loader.py): `services/worker/model_loader.py`

### Redis state

- **Redis**

### Other components

- **Inference client**
- [Container and Kubernetes configuration](https://github.com/rares8921/cost-aware-inference-cluster/tree/main/kubernetes): `kubernetes`
- [Prometheus and Grafana configuration](https://github.com/rares8921/cost-aware-inference-cluster/tree/main/monitoring): `monitoring`

## Connections

- Inference client → Router API: submits request (evidence: `README.md`)
- Router API → Tenant limits: checks quota (evidence: `services/router/main.py`)
- Router API → Scheduler API: forwards request (evidence: `services/router/main.py`)
- Tenant limits → Redis: reads and updates limits
- Scheduler API → Priority queue: enqueues request (evidence: `services/scheduler/main.py`)
- Scheduler API → Worker registry: records heartbeats (evidence: `services/scheduler/main.py`)
- Priority queue → Redis: stores queue state (evidence: `services/scheduler/queue_manager.py`)
- Worker registry → Redis: stores worker state (evidence: `services/scheduler/worker_registry.py`)
- Scheduler API → Autoscaler: starts decision loop (evidence: `services/scheduler/main.py`)
- Autoscaler → Priority queue: reads queue metrics (evidence: `services/scheduler/autoscaler.py`)
- Autoscaler → Worker registry: reads and removes workers (evidence: `services/scheduler/autoscaler.py`)
- Autoscaler → Cost optimizer: checks budget (evidence: `services/scheduler/autoscaler.py`)
- Worker service → Scheduler API: sends heartbeat (evidence: `services/worker/main.py`)
- Worker service → Scheduler API: polls for work (evidence: `services/worker/main.py`)
- Scheduler API → Priority queue: dequeues items (evidence: `services/scheduler/main.py`)
- Worker service → Dynamic batching: adds items (evidence: `services/worker/main.py`)
- Dynamic batching → Model inference: processes text batches (evidence: `README.md`)
- Worker service → Scheduler API: reports completion (evidence: `services/worker/main.py`)
- Scheduler API → Priority queue: records completion (evidence: `services/scheduler/main.py`)
- Container and Kubernetes configuration → Router API: configures service
- Container and Kubernetes configuration → Scheduler API: configures service
- Container and Kubernetes configuration → Worker service: configures service
- Prometheus and Grafana configuration → Scheduler API: scrapes and displays metrics
