# manassharma2006/llm-autoscaler architecture

> Architecture diagram of the GitHub repository manassharma2006/llm-autoscaler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/manassharma2006/llm-autoscaler
- Repository: https://github.com/manassharma2006/llm-autoscaler

## Overview

This is a heterogeneous LLM inference gateway with separate Fast and Heavy queues, Redis-backed worker tracking, and a fuzzy-logic autoscaling control plane. Clients submit prompts; the gateway selects a tier and enqueues work, workers call an inference backend and save results for polling. Separately, an evaluator combines queue, growth, latency, and worker signals, while a daemon actuates replicas through Docker Compose or Kubernetes. The README documents deployment modes and Ollama/Groq capabilities; those unsampled deployment and integration details are shown as documented relationships, not inspected implementation. Efficacy branches are limited to excerpted calls and explicit README statements.

Main entry points: [`fastapi-service/app/worker/worker.py`](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/worker/worker.py) (Tier Workers), [`fastapi-service/app/main.py`](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/main.py) (FastAPI App), [`fastapi-service/app/services/autoscaler.py`](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/autoscaler.py) (Tier Autoscaler), [`fastapi-service/run_autoscaler.py`](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/run_autoscaler.py) (Autoscaler Daemon)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_gateway["API Gateway"]
  node_api["FastAPI App<br/>[main.py]"]
  node_schemas["API Schemas<br/>[schemas.py]"]
  node_inference_route["Inference Routes<br/>[inference.py]"]
  node_worker_routes["Worker Routes<br/>[workers.py]"]
  node_autoscaler_routes["Autoscaler Routes<br/>[autoscaler.py]"]
  node_config["Runtime Settings<br/>[config.py]"]
end

subgraph group_execution["Inference Execution"]
  node_worker["Tier Workers<br/>[worker.py]"]
  node_ollama["Ollama Engine"]
  node_groq["Groq API"]
end

subgraph group_state["Redis State"]
  node_redis_service[("Queue and Results<br/>[redis_service.py]")]
  node_worker_registry[("Worker Registry<br/>[worker_registry.py]")]
end

subgraph group_control["Autoscaling Control"]
  node_autoscaler["Tier Autoscaler<br/>[autoscaler.py]"]
  node_fuzzy["Fuzzy Controller<br/>[fuzzy_scaler.py]"]
  node_daemon["Autoscaler Daemon<br/>[run_autoscaler.py]"]
end

subgraph group_platform["Scaling Backends"]
  node_docker["Docker Compose"]
  node_kubernetes["Kubernetes"]
end

node_client(("HTTP Client"))

node_client -->|"sends HTTP"| node_api
node_api -->|"registers routes"| node_inference_route
node_api -->|"registers routes"| node_worker_routes
node_api -->|"registers routes"| node_autoscaler_routes
node_inference_route -->|"validates and responds"| node_schemas
node_inference_route -->|"enqueues and reads"| node_redis_service
node_worker_routes -->|"lists and reads"| node_worker_registry
node_autoscaler_routes -->|"evaluates and reports"| node_autoscaler
node_worker -->|"claims and stores results"| node_redis_service
node_worker -->|"registers and heartbeats"| node_worker_registry
node_worker -.->|"generates with"| node_ollama
node_worker -.->|"optionally calls"| node_groq
node_autoscaler -->|"reads worker metrics"| node_worker_registry
node_autoscaler -->|"reads queue depth"| node_redis_service
node_autoscaler -->|"evaluates signals"| node_fuzzy
node_daemon -->|"requests evaluation"| node_autoscaler_routes
node_daemon -->|"scales replicas"| node_docker
node_daemon -->|"scales replicas"| node_kubernetes

click node_api "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/main.py"
click node_schemas "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/models/schemas.py"
click node_inference_route "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/inference.py"
click node_worker_routes "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/workers.py"
click node_autoscaler_routes "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/autoscaler.py"
click node_config "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/config.py"
click node_worker "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/worker/worker.py"
click node_redis_service "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/redis_service.py"
click node_worker_registry "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/worker_registry.py"
click node_autoscaler "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/autoscaler.py"
click node_fuzzy "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/fuzzy_scaler.py"
click node_daemon "https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/run_autoscaler.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_schemas,node_inference_route,node_worker_routes,node_autoscaler_routes,node_config,node_client toneBlue
class node_worker,node_ollama,node_groq toneAmber
class node_redis_service,node_worker_registry toneMint
class node_autoscaler,node_fuzzy,node_daemon toneRose
class node_docker,node_kubernetes toneIndigo
```

## Components

### API Gateway

- [FastAPI App](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/main.py): `fastapi-service/app/main.py`
- [API Schemas](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/models/schemas.py): `fastapi-service/app/models/schemas.py`
- [Inference Routes](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/inference.py): `fastapi-service/app/routes/inference.py`
- [Worker Routes](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/workers.py): `fastapi-service/app/routes/workers.py`
- [Autoscaler Routes](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/routes/autoscaler.py): `fastapi-service/app/routes/autoscaler.py`
- [Runtime Settings](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/config.py): `fastapi-service/app/config.py`

### Inference Execution

- [Tier Workers](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/worker/worker.py): `fastapi-service/app/worker/worker.py`
- **Ollama Engine**
- **Groq API**

### Redis State

- [Queue and Results](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/redis_service.py): `fastapi-service/app/services/redis_service.py`
- [Worker Registry](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/worker_registry.py): `fastapi-service/app/services/worker_registry.py`

### Autoscaling Control

- [Tier Autoscaler](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/autoscaler.py): `fastapi-service/app/services/autoscaler.py`
- [Fuzzy Controller](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/app/services/fuzzy_scaler.py): `fastapi-service/app/services/fuzzy_scaler.py`
- [Autoscaler Daemon](https://github.com/manassharma2006/llm-autoscaler/blob/main/fastapi-service/run_autoscaler.py): `fastapi-service/run_autoscaler.py`

### Scaling Backends

- **Docker Compose**
- **Kubernetes**

### Other components

- **HTTP Client**

## Connections

- HTTP Client → FastAPI App: sends HTTP (evidence: `README.md`)
- FastAPI App → Inference Routes: registers routes (evidence: `fastapi-service/app/main.py`)
- FastAPI App → Worker Routes: registers routes (evidence: `fastapi-service/app/main.py`)
- FastAPI App → Autoscaler Routes: registers routes (evidence: `fastapi-service/app/main.py`)
- Inference Routes → API Schemas: validates and responds (evidence: `fastapi-service/app/routes/inference.py`)
- Inference Routes → Queue and Results: enqueues and reads (evidence: `fastapi-service/app/routes/inference.py`)
- Worker Routes → Worker Registry: lists and reads (evidence: `fastapi-service/app/routes/workers.py`)
- Autoscaler Routes → Tier Autoscaler: evaluates and reports (evidence: `fastapi-service/app/routes/autoscaler.py`)
- Tier Workers → Queue and Results: claims and stores results (evidence: `README.md`)
- Tier Workers → Worker Registry: registers and heartbeats (evidence: `fastapi-service/app/worker/worker.py`)
- Tier Workers → Ollama Engine: generates with (evidence: `README.md`)
- Tier Workers → Groq API: optionally calls (evidence: `README.md`)
- Tier Autoscaler → Worker Registry: reads worker metrics (evidence: `fastapi-service/app/services/autoscaler.py`)
- Tier Autoscaler → Queue and Results: reads queue depth (evidence: `fastapi-service/app/services/autoscaler.py`)
- Tier Autoscaler → Fuzzy Controller: evaluates signals (evidence: `fastapi-service/app/services/autoscaler.py`)
- Autoscaler Daemon → Autoscaler Routes: requests evaluation (evidence: `fastapi-service/run_autoscaler.py`)
- Autoscaler Daemon → Docker Compose: scales replicas (evidence: `README.md`)
- Autoscaler Daemon → Kubernetes: scales replicas (evidence: `README.md`)
