How rares8921/cost-aware-inference-cluster works
This prototype accepts inference requests through a tenant-aware router, queues them at the scheduler, and has workers poll, batch, and run model inference before reporting completion. The scheduler also tracks worker heartbeats and makes cost-aware scaling decisions; the README says those decisions are logged, not enacted by creating pods or containers. Redis is shared by rate-limit, queue, and worker-registry state. The graph focuses on runtime paths and documented capabilities; unsampled configuration and monitoring/deployment wiring are not expanded into unverified runtime links.
Main entry points: scheduler/main.py Scheduler API · worker/main.py Worker service · queue_manager.py Priority queue · autoscaler.py Autoscaler
Components 14
Request routing
- Router API
services/router/main.py - Tenant limits
services/router/tenant_manager.py
Scheduling and scaling
- Scheduler API
services/scheduler/main.py - Priority queue
services/scheduler/queue_manager.py - Worker registry
services/scheduler/worker_registry.py - Autoscaler
services/scheduler/autoscaler.py - Cost optimizer
services/scheduler/cost_optimizer.py
Worker inference
- Worker service
services/worker/main.py - Dynamic batching
services/worker/batch_processor.py - Model inference
services/worker/model_loader.py
Redis state
- Redis
Other components
- Inference client
- Container and Kubernetes configuration
kubernetes - Prometheus and Grafana configuration
monitoring
Connections 23
- Inference client → Router API: submits request
README.md - Router API → Tenant limits: checks quota
services/router/main.py - Router API → Scheduler API: forwards request
services/router/main.py - Tenant limits → Redis: reads and updates limits
- Scheduler API → Priority queue: enqueues request
services/scheduler/main.py - Scheduler API → Worker registry: records heartbeats
services/scheduler/main.py - Priority queue → Redis: stores queue state
services/scheduler/queue_manager.py - Worker registry → Redis: stores worker state
services/scheduler/worker_registry.py - Scheduler API → Autoscaler: starts decision loop
services/scheduler/main.py - Autoscaler → Priority queue: reads queue metrics
services/scheduler/autoscaler.py - Autoscaler → Worker registry: reads and removes workers
services/scheduler/autoscaler.py - Autoscaler → Cost optimizer: checks budget
services/scheduler/autoscaler.py - Worker service → Scheduler API: sends heartbeat
services/worker/main.py - Worker service → Scheduler API: polls for work
services/worker/main.py - Scheduler API → Priority queue: dequeues items
services/scheduler/main.py - Worker service → Dynamic batching: adds items
services/worker/main.py - Dynamic batching → Model inference: processes text batches
README.md - Worker service → Scheduler API: reports completion
services/worker/main.py - Scheduler API → Priority queue: records completion
services/scheduler/main.py - Container and Kubernetes configuration → Router API: configures service
- Container and Kubernetes configuration → Scheduler API: configures service
- Container and Kubernetes configuration → Worker service: configures service
- Prometheus and Grafana configuration → Scheduler API: scrapes and displays metrics