# gorizond/koldun architecture

> Architecture diagram of the GitHub repository gorizond/koldun, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gorizond/koldun
- Repository: https://github.com/gorizond/koldun

## Overview

##### Project type and purpose
- This is a Kubernetes Operator project, written in Go, that orchestrates distributed-llama inference topologies on Kubernetes.
- It exposes four runtime “modes” via a single binary (`cmd/operator`):
- operator (CRD controllers)
- ingress (OpenAI-compatible HTTP API)
- dispatcher (NATS backlog consumer & assignment writer)
- llm (worker sidecar streaming completions)
- It uses custom resources (CRDs) – Session, Dllama, Model, Root, Worker, Ingress – plus NATS JetStream for messaging and MinIO/S3 (CSI) for model storage.

##### File structure and tech stack
- Top-level directories:
- cmd/operator – binary entrypoint wiring CLI → chosen mode
- pkg/apis – Kubernetes CRD type definitions
- pkg/controllers – Wrangler reconcilers for each CRD
- pkg/servers/{ingress,dispatcher,llm,operator} – HTTP servers & workers per mode
- charts/koldun, k8s/ – Helm chart & raw Kubernetes manifests
- hack, .github/workflows – CI, E2E test scripts, coverage helpers
- Key external systems: Kubernetes API, NATS JetStream, MinIO/S3 CSI, Helm/Skaffold for deployment
- Patterns: Operator (reconcile loops), event-driven via JetStream, microservices pods per mode, Helm for packaging

##### Core components
- Clients
- HTTP clients (OpenAI-compatible)
- Ingress Backend
- Ingress HTTP server pod (mode=ingress)
- AuthN/Z, conversation manifest → JetStream KV “backend-conversation-bucket”
- NATS JetStream cluster
- Streams & KV buckets: conversations, assignments, models, tokens
- Kubernetes Operator
- Operator pod (mode=operator) running controllers
- Watches JetStream KV and CRDs, reconciles Session/Dllama/Model/Ingress resources
- Dispatcher
- Dispatcher pod(s) (mode=dispatcher)
- Reads sessions.<hash>.requests, writes assignments into JetStream KV
- Dllama Topology
- Root coordinator Deployment & Service
- Worker StatefulSets (one per slot)
- LLM Worker sidecar
- llm pod(s) streaming completion chunks from dllama-api → NATS out.<hash>
- Heartbeat/state subjects for dispatcher
- Object Storage
- MinIO/S3 CSI PV/PVC for model artifacts
- GGUF converter jobs that run as Kubernetes Jobs under the Model CR
- Deployment & Packaging
- Helm chart, raw manifests, Skaffold for builds
- GitHub Actions & local docker-compose for CI/E2E

##### Relationships & data flows
- Client → Ingress HTTP API → JetStream KV (conversation manifest)
- JetStream KV → Operator reconciler → Session + Dllama CRs → Kubernetes API
- Operator → creates Deployments/StatefulSets/Jobs for Model, Root, Worker, Ingress CRs
- Ingress Backend → publishes chat requests to NATS subjects (sessions.<hash>.requests)
- Dispatcher → consumes requests, assigns to Worker pods via assignments KV
- Worker (llm) → streams completion chunks to NATS out.<hash> → consumed by client or dispatcher for backpressure
- Model CR → kicks off Kubernetes Jobs (download, convert) → writes size metadata back to CR status
- Helm/Skaffold dots → install CRDs, operator, NATS, MinIO, sample CRs

##### Architectural patterns & design principles
- Kubernetes Operator pattern (CRD + reconciliation loops)
- Event-driven messaging via NATS JetStream and KV stores
- Mode‐based microservices: one binary, different entrypoint modes running in separate pods
- Separation of concerns: API ingestion, orchestration, dispatch, model management, LLM execution
- Helm chart for declarative packaging, Compose for local integration testing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        Client(("HTTP Client")):::client
        NATS[(NATS JetStream Cluster)]:::messaging
        MinIO[(MinIO/S3 CSI)]:::storage
    end

    %% Kubernetes Cluster
    subgraph "Kubernetes Cluster"
        subgraph "Control Plane & Operator"
            OperatorPod[/"Operator Pod (mode=operator)"/]:::operator
            Controllers["Reconcilers (pkg/controllers)"]:::operator
            subgraph "Custom Resources (CRDs)"
                SessionCR["Session CRD"]:::crd
                DllamaCR["Dllama CRD"]:::crd
                ModelCR["Model CRD"]:::crd
                RootCR["Root CRD"]:::crd
                WorkerCR["Worker CRD"]:::crd
                IngressCR["Ingress CRD"]:::crd
            end
        end

        subgraph "Runtime Pods"
            IngressPod[/"Ingress Pod (mode=ingress)"/]:::runtime
            DispatcherPod[/"Dispatcher Pod (mode=dispatcher)"/]:::runtime
            WorkerPod[/"LLM Worker Sidecar (mode=llm)"/]:::runtime
        end

        subgraph "Dllama Topology"
            RootSvc["Root Service & Deployment"]:::runtime
            WorkerSTS["Worker StatefulSet"]:::runtime
        end

        subgraph "Model Storage Jobs"
            DownloadJob["Model Download Job"]:::job
            ConvertJob["GGUF Conversion Job"]:::job
        end
    end

    %% CI/CD & Deployment
    subgraph "CI/CD & Deployment"
        HelmChart(("Helm Chart")):::cicd
        Manifests(("Raw k8s Manifests")):::cicd
        Skaffold(("Skaffold Config")):::cicd
        GitHubActions(("GitHub Actions")):::cicd
        DockerCompose(("docker-compose.test.yml")):::cicd
    end

    %% Relationships
    Client -->|"HTTP"| IngressPod
    IngressPod -->|"KV: conversations"| NATS
    NATS -->|"watch KV & streams"| OperatorPod
    OperatorPod -->|"uses reconcilers"| Controllers
    Controllers -->|"manage"| SessionCR
    Controllers -->|"manage"| DllamaCR
    Controllers -->|"manage"| ModelCR
    Controllers -->|"manage"| RootCR
    Controllers -->|"manage"| WorkerCR
    Controllers -->|"manage"| IngressCR

    Controllers -->|"creates"| RootSvc
    Controllers -->|"creates"| WorkerSTS
    Controllers -->|"creates"| IngressPod
    Controllers -->|"creates"| DownloadJob
    Controllers -->|"creates"| ConvertJob

    IngressPod -->|"publish requests"| NATS
    DispatcherPod -->|"consume requests"| NATS
    DispatcherPod -->|"KV: assignments"| NATS
    WorkerPod -->|"stream chunks"| NATS

    ModelCR -->|"triggers"| DownloadJob
    DownloadJob -->|"updates status"| ModelCR
    ConvertJob -->|"updates status"| ModelCR

    HelmChart -->|"install CRDs & Operator & NATS & MinIO"| KubernetesCluster
    Skaffold -->|"build & deploy"| KubernetesCluster
    GitHubActions -->|"CI/E2E tests"| KubernetesCluster
    DockerCompose -->|"local integration test"| KubernetesCluster
    Manifests -->|"apply raw YAML"| KubernetesCluster

    %% Click Events
    click OperatorPod "https://github.com/gorizond/koldun/blob/main/cmd/operator/main.go"
    click OperatorPod "https://github.com/gorizond/koldun/blob/main/pkg/servers/operator/server.go"
    click SessionCR "https://github.com/gorizond/koldun/blob/main/pkg/apis/koldun.gorizond.io/v1/types.go"
    click SessionCR "https://github.com/gorizond/koldun/blob/main/charts/koldun/templates/crd/bases/*.yaml"
    click Controllers "https://github.com/gorizond/koldun/tree/main/pkg/controllers"
    click IngressPod "https://github.com/gorizond/koldun/blob/main/pkg/servers/ingress/server.go"
    click DispatcherPod "https://github.com/gorizond/koldun/blob/main/pkg/servers/dispatcher/server.go"
    click WorkerPod "https://github.com/gorizond/koldun/blob/main/pkg/servers/llm/server.go"
    click HelmChart "https://github.com/gorizond/koldun/blob/main/charts/koldun/Chart.yaml"
    click Manifests "https://github.com/gorizond/koldun/blob/main/k8s/*.yaml"
    click Skaffold "https://github.com/gorizond/koldun/blob/main/skaffold.yaml"
    click GitHubActions "https://github.com/gorizond/koldun/blob/main/.github/workflows/*.yaml"
    click DockerCompose "https://github.com/gorizond/koldun/blob/main/docker-compose.test.yml"

    %% Styles
    classDef client fill:#cdeeee,stroke:#000;
    classDef messaging fill:#bbdefb,stroke:#000;
    classDef storage fill:#ffe0b2,stroke:#000;
    classDef operator fill:#c8e6c9,stroke:#000;
    classDef runtime fill:#d1c4e9,stroke:#000;
    classDef crd fill:#e1bee7,stroke:#000;
    classDef job fill:#ffcdd2,stroke:#000;
    classDef cicd fill:#eeeeee,stroke:#000;
```
