How gorizond/koldun works
Project type and purpose
• This is a Kubernetes Operator project, written in Go, that orchestrates distributed-llama inference topologies on Kubernetes.
• It exposes four runtime “modes” via a single binary (cmd/operator):
• operator (CRD controllers)
• ingress (OpenAI-compatible HTTP API)
• dispatcher (NATS backlog consumer & assignment writer)
• llm (worker sidecar streaming completions)
• It uses custom resources (CRDs) – Session, Dllama, Model, Root, Worker, Ingress – plus NATS JetStream for messaging and MinIO/S3 (CSI) for model storage.
File structure and tech stack
• Top-level directories:
• cmd/operator – binary entrypoint wiring CLI → chosen mode
• pkg/apis – Kubernetes CRD type definitions
• pkg/controllers – Wrangler reconcilers for each CRD
• pkg/servers/{ingress,dispatcher,llm,operator} – HTTP servers & workers per mode
• charts/koldun, k8s/ – Helm chart & raw Kubernetes manifests
• hack, .github/workflows – CI, E2E test scripts, coverage helpers
• Key external systems: Kubernetes API, NATS JetStream, MinIO/S3 CSI, Helm/Skaffold for deployment
• Patterns: Operator (reconcile loops), event-driven via JetStream, microservices pods per mode, Helm for packaging
Core components
• Clients
• HTTP clients (OpenAI-compatible)
• Ingress Backend
• Ingress HTTP server pod (mode=ingress)
• AuthN/Z, conversation manifest → JetStream KV “backend-conversation-bucket”
• NATS JetStream cluster
• Streams & KV buckets: conversations, assignments, models, tokens
• Kubernetes Operator
• Operator pod (mode=operator) running controllers
• Watches JetStream KV and CRDs, reconciles Session/Dllama/Model/Ingress resources
• Dispatcher
• Dispatcher pod(s) (mode=dispatcher)
• Reads sessions.<hash>.requests, writes assignments into JetStream KV
• Dllama Topology
• Root coordinator Deployment & Service
• Worker StatefulSets (one per slot)
• LLM Worker sidecar
• llm pod(s) streaming completion chunks from dllama-api → NATS out.<hash>
• Heartbeat/state subjects for dispatcher
• Object Storage
• MinIO/S3 CSI PV/PVC for model artifacts
• GGUF converter jobs that run as Kubernetes Jobs under the Model CR
• Deployment & Packaging
• Helm chart, raw manifests, Skaffold for builds
• GitHub Actions & local docker-compose for CI/E2E
Relationships & data flows
• Client → Ingress HTTP API → JetStream KV (conversation manifest)
• JetStream KV → Operator reconciler → Session + Dllama CRs → Kubernetes API
• Operator → creates Deployments/StatefulSets/Jobs for Model, Root, Worker, Ingress CRs
• Ingress Backend → publishes chat requests to NATS subjects (sessions.<hash>.requests)
• Dispatcher → consumes requests, assigns to Worker pods via assignments KV
• Worker (llm) → streams completion chunks to NATS out.<hash> → consumed by client or dispatcher for backpressure
• Model CR → kicks off Kubernetes Jobs (download, convert) → writes size metadata back to CR status
• Helm/Skaffold dots → install CRDs, operator, NATS, MinIO, sample CRs
Architectural patterns & design principles
• Kubernetes Operator pattern (CRD + reconciliation loops)
• Event-driven messaging via NATS JetStream and KV stores
• Mode‐based microservices: one binary, different entrypoint modes running in separate pods
• Separation of concerns: API ingestion, orchestration, dispatch, model management, LLM execution
• Helm chart for declarative packaging, Compose for local integration testing