How manassharma2006/llm-autoscaler works
This is a heterogeneous LLM inference gateway with separate Fast and Heavy queues, Redis-backed worker tracking, and a fuzzy-logic autoscaling control plane. Clients submit prompts; the gateway selects a tier and enqueues work, workers call an inference backend and save results for polling. Separately, an evaluator combines queue, growth, latency, and worker signals, while a daemon actuates replicas through Docker Compose or Kubernetes. The README documents deployment modes and Ollama/Groq capabilities; those unsampled deployment and integration details are shown as documented relationships, not inspected implementation. Efficacy branches are limited to excerpted calls and explicit README statements.
Main entry points: worker.py Tier Workers · main.py FastAPI App · autoscaler.py Tier Autoscaler · run_autoscaler.py Autoscaler Daemon
Components 17
API Gateway
- FastAPI App
fastapi-service/app/main.py - API Schemas
fastapi-service/app/models/schemas.py - Inference Routes
fastapi-service/app/routes/inference.py - Worker Routes
fastapi-service/app/routes/workers.py - Autoscaler Routes
fastapi-service/app/routes/autoscaler.py - Runtime Settings
fastapi-service/app/config.py
Inference Execution
- Tier Workers
fastapi-service/app/worker/worker.py - Ollama Engine
- Groq API
Redis State
- Queue and Results
fastapi-service/app/services/redis_service.py - Worker Registry
fastapi-service/app/services/worker_registry.py
Autoscaling Control
- Tier Autoscaler
fastapi-service/app/services/autoscaler.py - Fuzzy Controller
fastapi-service/app/services/fuzzy_scaler.py - Autoscaler Daemon
fastapi-service/run_autoscaler.py
Scaling Backends
- Docker Compose
- Kubernetes
Other components
- HTTP Client
Connections 18
- HTTP Client → FastAPI App: sends HTTP
README.md - FastAPI App → Inference Routes: registers routes
fastapi-service/app/main.py - FastAPI App → Worker Routes: registers routes
fastapi-service/app/main.py - FastAPI App → Autoscaler Routes: registers routes
fastapi-service/app/main.py - Inference Routes → API Schemas: validates and responds
fastapi-service/app/routes/inference.py - Inference Routes → Queue and Results: enqueues and reads
fastapi-service/app/routes/inference.py - Worker Routes → Worker Registry: lists and reads
fastapi-service/app/routes/workers.py - Autoscaler Routes → Tier Autoscaler: evaluates and reports
fastapi-service/app/routes/autoscaler.py - Tier Workers → Queue and Results: claims and stores results
README.md - Tier Workers → Worker Registry: registers and heartbeats
fastapi-service/app/worker/worker.py - Tier Workers → Ollama Engine: generates with
README.md - Tier Workers → Groq API: optionally calls
README.md - Tier Autoscaler → Worker Registry: reads worker metrics
fastapi-service/app/services/autoscaler.py - Tier Autoscaler → Queue and Results: reads queue depth
fastapi-service/app/services/autoscaler.py - Tier Autoscaler → Fuzzy Controller: evaluates signals
fastapi-service/app/services/autoscaler.py - Autoscaler Daemon → Autoscaler Routes: requests evaluation
fastapi-service/run_autoscaler.py - Autoscaler Daemon → Docker Compose: scales replicas
README.md - Autoscaler Daemon → Kubernetes: scales replicas
README.md