# cybozu-go/cke architecture

> Architecture diagram of the GitHub repository cybozu-go/cke, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cybozu-go/cke
- Repository: https://github.com/cybozu-go/cke

## Overview

##### Project Type & Purpose
- This is a full-blown distributed control plane for Kubernetes cluster management (Cybozu Kubernetes Engine).
- It comprises three primary programs:
– `cke`: the main control-plane service that orchestrates bootstrapping, upgrades, reboots, repairs.
– `ckecli`: a CLI “frontend” to drive the `cke` service via its HTTP API.
– `cke-localproxy`: an optional per-node sidecar that proxies Kubernetes API calls (e.g., kube-proxy) and mediates certain operations.
- It integrates with external systems: etcd for config storage & leader election, Vault for secrets/certificates, Sabakan for hardware inventory, and the target Kubernetes cluster API.

##### High-Level File-Structure Analysis
- Top-level Go packages & binaries:
– `agent.go`, `cluster.go`, `container.go`, etc. define core data models & orchestrator logic.
– `server/`: HTTP API server handlers (`control.go`, `watch.go`, `strategy.go`).
– `op/`: a library of “operation” phases (etcd ops, k8s ops, reboot/repair steps).
– `sabakan/`: Sabakan integration (config generation, GraphQL client, repairer).
– `metrics/`: collection & updating of runtime metrics.
– `localproxy/`: the localproxy entrypoint, operation & status commands.
– `pkg/ckecli/`: the CLI command definitions using Cobra.
– `server/`, `op/`, `sabakan/`, `metrics/`, `localproxy/`, `pkg/ckecli/` each correspond to logical subcomponents.
- Auxiliary directories:
– `docs/`, `example/`, `sonobuoy/`, `mtest/` for documentation, examples, smoketests.
– `docker/`, `tools/` for build/deployment tooling.

##### Key Architectural Insights from README
- Distributed multi-instance service: uses etcd for shared config, leader election & operation logs.
- Certificates: CKE can spin up its own CA via Vault integration.
- Life-cycle management: orchestrates Kubernetes and etcd clusters via SSH + Docker or CRI.
- Pluggable operations: each operation (upgrade, reboot, repair) is implemented as a sequence of op modules in `op/`.
- Sabakan integration is optional but first-class.

a. Main Components
- User Interface (CLI): `ckecli`
- Local Sidecar: `cke-localproxy` on each node
- Control-Plane Service: `cke`
- Operation Engine: `op/` modules invoked by the service
- Persistence Layer: etcd cluster
- Secrets & PKI: Vault server
- Target Kubernetes: API server + kubelets + CNI + CoreDNS on the managed cluster
- Sabakan (optional): hardware provisioning service

b. Component Interactions
- User → `ckecli`: invokes commands (e.g., `ckecli cluster set`)
- `ckecli` → HTTP API → `cke`
- `cke` ↔ etcd: read cluster config, write operation logs, leader election
- `cke` ↔ Vault: generate & store certificates, encryption keys
- `cke` → SSH / Docker (or CRI) → worker & control-plane nodes to deploy containers & kube components
- `cke` → Kubernetes API (direct or via `cke-localproxy`): deploy or patch k8s resources, taints, eviction calls for reboot
- `cke` → sabakan: fetch machine inventory or trigger repairs

c. Architectural Patterns & Principles
- Leader-election & shared state via distributed datastore (etcd).
- Command-pattern for CLI commands (Cobra in `pkg/ckecli`).
- Plugin/phase-based engine (`op/` modules) enabling easy extension.
- Separation of concerns: CLI vs. API service vs. sidecar vs. op logic vs. persistence.

d. Key Technologies & Frameworks
- Go modules & standard library (SSH, HTTP, JSON).
- Cobra for CLI.
- gRPC or HTTP+JSON for internal API (HTTP in `server/`).
- etcd v3 client for coordination & storage.
- Vault API for PKI.
- Docker / CRI for node-level container orchestration.
- GraphQL client for Sabakan (`sabakan/query.go`).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User & Tools Layer
    subgraph "User & DevOps Tools"
        direction TB
        CLI["CLI (ckecli)"]:::cli
        Sabakan_UI["Sabakan UI (optional)"]:::external
    end

    %% Control-Plane Service Layer
    subgraph "Control-Plane Service (cke)"
        direction TB
        API["CKE API Server"]:::cp
        Engine["Operation Engine"]:::cp
        Sabakan_Conn["Sabakan Connector"]:::cp
        Metrics["Metrics Collector"]:::cp
        Etcd_Client["etcd Client"]:::storage
        Vault_Client["Vault Client"]:::storage
    end

    %% Managed Cluster Layer
    subgraph "Managed Cluster"
        direction TB
        etcd_cluster["etcd Cluster"]:::storage
        Vault_Server["Vault Server"]:::storage
        subgraph "Cluster Node Pool"
            direction TB
            K8s_API["Kubernetes API Server"]:::external
            kubelet["kubelet + CNI + CoreDNS"]:::external
            LocalProxy["Local Sidecar (cke-localproxy)"]:::cli
        end
    end

    %% Connections
    CLI -->|"HTTP/JSON"| API
    Sabakan_UI -.->|"GraphQL"| Sabakan_Conn
    API -->|"invoke ops"| Engine
    Engine -->|"read/write config"| Etcd_Client
    Etcd_Client -->|"watch logs"| Engine
    Engine -->|"Vault API"| Vault_Client
    Vault_Client --> Vault_Server
    Engine -->|"GraphQL"| Sabakan_Conn
    Engine -->|"SSH/Docker/CRI"| kubelet
    Engine -->|"K8s API"| K8s_API
    etcd_cluster <--> Etcd_Client

    %% Click Events
    click CLI "https://github.com/cybozu-go/cke/blob/main/pkg/ckecli/main.go"
    click API "https://github.com/cybozu-go/cke/tree/main/server/"
    click Engine "https://github.com/cybozu-go/cke/tree/main/op/"
    click Sabakan_Conn "https://github.com/cybozu-go/cke/tree/main/sabakan/"
    click Metrics "https://github.com/cybozu-go/cke/blob/main/metrics/metrics.go"
    click Etcd_Client "https://github.com/cybozu-go/cke/blob/main/storage.go"
    click Vault_Client "https://github.com/cybozu-go/cke/blob/main/vault.go"
    click LocalProxy "https://github.com/cybozu-go/cke/blob/main/pkg/cke-localproxy/main.go"

    %% Styles
    classDef cli fill:#b3e6ff,stroke:#333,stroke-width:1px
    classDef cp fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef external fill:#ffe6cc,stroke:#333,stroke-width:1px
    classDef storage fill:#ffd699,stroke:#333,stroke-width:1px
```
