How scaleapi/llm-engine works
Purpose and entry points
LLM Engine is a Python client plus a self-hostable Kubernetes control plane for fine-tuning, deploying, and serving LLMs. Hosted users enter through the Python SDK (clients/python/llmengine/api_engine.py, clients/python/llmengine/completion.py). Self-hosted control-plane processes enter through FastAPI and operational scripts: model-engine/model_engine_server/entrypoints/start_fastapi_server.py, model-engine/model_engine_server/entrypoints/start_batch_job_orchestration.py, and model-engine/model_engine_server/entrypoints/init_database.py.
API boundary and application composition
The FastAPI app exposes versioned REST resources for LLMs, endpoints, bundles, files, jobs, tasks, and triggers. API handlers translate HTTP DTOs into domain use cases; dependency wiring chooses live or fake/configured implementations. OpenAI-compatible completion/chat/batch routes are separated under v2.
Paths: model-engine/model_engine_server/api/app.py, model-engine/model_engine_server/api/model_endpoints_v1.py, model-engine/model_engine_server/api/v2/chat_completion.py.
Domain orchestration layer
Domain entities model endpoints, model bundles, LLMs/fine-tunes, files, and batch jobs. Use cases coordinate authorization, persistence, endpoint lifecycle, inference, streaming, fine-tuning, and scheduled triggers behind gateway interfaces. This is the principal control boundary: domain code depends on abstract gateways/repositories, while infrastructure supplies cloud/Kubernetes implementations.
Paths: model-engine/model_engine_server/domain/use_cases/model_endpoint_use_cases.py, model-engine/model_engine_server/domain/use_cases/sync_inference_use_cases.py, model-engine/model_engine_server/domain/gateways/sync_model_endpoint_inference_gateway.py.
Endpoint provisioning and lifecycle
Endpoint creation resolves model/bundle metadata and provisions Kubernetes-backed serving resources through a live infrastructure gateway. Resource delegates support direct Kubernetes endpoints and queue-backed endpoint variants; cache/image-cache services support faster startup. The Helm deployment includes dedicated endpoint-builder and cacher workloads.
Paths: model-engine/model_engine_server/infra/services/live_endpoint_builder_service.py, model-engine/model_engine_server/infra/gateways/live_model_endpoint_infra_gateway.py, model-engine/model_engine_server/infra/gateways/resources/k8s_endpoint_resource_delegate.py.
Inference execution paths
Synchronous, streaming, and asynchronous inference are distinct paths from API/domain gateways to deployed endpoint services. Serving images and wrappers support vLLM, SGLang, TensorRT-LLM/Triton, generic HTTP forwarding, and Celery forwarding. Async work is queued through Celery; batch inference has vLLM/Ray-oriented execution assets.
Paths: model-engine/model_engine_server/inference/sync_inference/fastapi_server.py, model-engine/model_engine_server/inference/vllm/vllm_server.py, model-engine/model_engine_server/inference/async_inference/tasks.py.
Persistence, queues, and external integrations
PostgreSQL stores control-plane records and schema migrations. Redis supports caches, feature flags, and autoscaling metrics; Celery provides task-queue execution. Artifact/file abstractions select S3 or GCS implementations; container registries include ECR, GAR, and ACR. Datadog provides monitoring, while AWS/GCP secret and role adapters isolate cloud specifics.
Paths: model-engine/model_engine_server/db/base.py, model-engine/model_engine_server/infra/gateways/s3_llm_artifact_gateway.py, model-engine/model_engine_server/infra/gateways/gcs_llm_artifact_gateway.py.
Deployment and operations
Helm is the self-hosting integration boundary: it deploys gateway/control-plane workloads, endpoint builder, cache, Celery autoscaler, database initialization/migration jobs, RBAC, autoscaling, and optional Istio routing. A separate startup watcher observes Kubernetes startup behavior. Dockerfiles package control-plane and inference runtimes.
Paths: charts/model-engine/templates/gateway_deployment.yaml, charts/model-engine/templates/endpoint_builder_deployment.yaml, charts/k8s-startup-watcher/templates/deployment.yaml.
Components 20
Client access
- Python SDK · Hosted client entry point for completions.
clients/python/llmengine/api_engine.py - Completion API
clients/python/llmengine/completion.py
Control plane
- FastAPI app · Versioned control-plane HTTP boundary.
model-engine/model_engine_server/api/app.py - v1 resources · LLM, endpoint, bundle, file, job, task, and trigger routes.
model-engine/model_engine_server/api/model_endpoints_v1.py - OpenAI-compatible v2 · Chat, completion, and batch-compatible routes.
model-engine/model_engine_server/api/v2/chat_completion.py - Domain use cases · Coordinates authorization, lifecycle, inference, fine-tuning, and triggers through abstractions.
model-engine/model_engine_server/domain/use_cases/model_endpoint_use_cases.py - Sync inference use cases
model-engine/model_engine_server/domain/use_cases/sync_inference_use_cases.py - Inference gateway contract · Abstract boundary for endpoint inference.
model-engine/model_engine_server/domain/gateways/sync_model_endpoint_inference_gateway.py - Live endpoint gateway · Cloud and Kubernetes implementation of endpoint lifecycle.
model-engine/model_engine_server/infra/gateways/live_model_endpoint_infra_gateway.py
Serving and execution
- Endpoint builder · Resolves model metadata and creates serving resources.
model-engine/model_engine_server/infra/services/live_endpoint_builder_service.py - Kubernetes endpoint delegate · Builds direct Kubernetes endpoint resources.
model-engine/model_engine_server/infra/gateways/resources/k8s_endpoint_resource_delegate.py - Sync inference server · Synchronous and streaming endpoint execution.
model-engine/model_engine_server/inference/sync_inference/fastapi_server.py - vLLM server · One of several model-serving backends.
model-engine/model_engine_server/inference/vllm/vllm_server.py - Async inference tasks · Queued asynchronous inference execution.
model-engine/model_engine_server/inference/async_inference/tasks.py
Platform integrations
- PostgreSQL · Stores records and migrations.
model-engine/model_engine_server/db/base.py - Artifact storage · S3 implementation selected behind file/artifact abstractions.
model-engine/model_engine_server/infra/gateways/s3_llm_artifact_gateway.py - GCS artifact storage · Alternative GCS artifact implementation.
model-engine/model_engine_server/infra/gateways/gcs_llm_artifact_gateway.py
Deployment
- Gateway deployment · Deploys the control-plane gateway workload.
charts/model-engine/templates/gateway_deployment.yaml - Endpoint builder deployment · Deploys dedicated endpoint provisioning workers.
charts/model-engine/templates/endpoint_builder_deployment.yaml - Kubernetes startup watcher · Observes Kubernetes startup behavior.
charts/k8s-startup-watcher/templates/deployment.yaml
Connections 21
- Python SDK → Completion API: uses
- Completion API → FastAPI app: calls
- FastAPI app → v1 resources: routes v1
- FastAPI app → OpenAI-compatible v2: routes v2
- v1 resources → Domain use cases: translates DTOs
- OpenAI-compatible v2 → Sync inference use cases: invokes
- Domain use cases → Sync inference use cases: coordinates inference
- Domain use cases → Live endpoint gateway: uses gateway
- Domain use cases → PostgreSQL: persists records
- Domain use cases → Artifact storage: stores artifacts
- Domain use cases → GCS artifact storage: alternative backend
- Sync inference use cases → Inference gateway contract: uses contract
- Inference gateway contract → Sync inference server: implemented by
- Live endpoint gateway → Endpoint builder: provisions through
- Endpoint builder → Kubernetes endpoint delegate: creates resources
- Kubernetes endpoint delegate → Sync inference server: deploys endpoint
- Sync inference server → vLLM server: serves models with
- Domain use cases → Async inference tasks: queues async work
- Gateway deployment → FastAPI app: deploys
- Endpoint builder deployment → Endpoint builder: deploys
- Kubernetes startup watcher → Gateway deployment: watches startup