How cybozu-go/cke works
Project Type & Purpose
• This is a full-blown distributed control plane for Kubernetes cluster management (Cybozu Kubernetes Engine).
• It comprises three primary programs:
– cke: the main control-plane service that orchestrates bootstrapping, upgrades, reboots, repairs.
– ckecli: a CLI “frontend” to drive the cke service via its HTTP API.
– cke-localproxy: an optional per-node sidecar that proxies Kubernetes API calls (e.g., kube-proxy) and mediates certain operations.
• It integrates with external systems: etcd for config storage & leader election, Vault for secrets/certificates, Sabakan for hardware inventory, and the target Kubernetes cluster API.
High-Level File-Structure Analysis
• Top-level Go packages & binaries:
– agent.go, cluster.go, container.go, etc. define core data models & orchestrator logic.
– server/: HTTP API server handlers (control.go, watch.go, strategy.go).
– op/: a library of “operation” phases (etcd ops, k8s ops, reboot/repair steps).
– sabakan/: Sabakan integration (config generation, GraphQL client, repairer).
– metrics/: collection & updating of runtime metrics.
– localproxy/: the localproxy entrypoint, operation & status commands.
– pkg/ckecli/: the CLI command definitions using Cobra.
– server/, op/, sabakan/, metrics/, localproxy/, pkg/ckecli/ each correspond to logical subcomponents.
• Auxiliary directories:
– docs/, example/, sonobuoy/, mtest/ for documentation, examples, smoketests.
– docker/, tools/ for build/deployment tooling.
Key Architectural Insights from README
• Distributed multi-instance service: uses etcd for shared config, leader election & operation logs.
• Certificates: CKE can spin up its own CA via Vault integration.
• Life-cycle management: orchestrates Kubernetes and etcd clusters via SSH + Docker or CRI.
• Pluggable operations: each operation (upgrade, reboot, repair) is implemented as a sequence of op modules in op/.
• Sabakan integration is optional but first-class.
a. Main Components
• User Interface (CLI): ckecli
• Local Sidecar: cke-localproxy on each node
• Control-Plane Service: cke
• Operation Engine: op/ modules invoked by the service
• Persistence Layer: etcd cluster
• Secrets & PKI: Vault server
• Target Kubernetes: API server + kubelets + CNI + CoreDNS on the managed cluster
• Sabakan (optional): hardware provisioning service
b. Component Interactions
• User → ckecli: invokes commands (e.g., ckecli cluster set)
• ckecli → HTTP API → cke
• cke ↔ etcd: read cluster config, write operation logs, leader election
• cke ↔ Vault: generate & store certificates, encryption keys
• cke → SSH / Docker (or CRI) → worker & control-plane nodes to deploy containers & kube components
• cke → Kubernetes API (direct or via cke-localproxy): deploy or patch k8s resources, taints, eviction calls for reboot
• cke → sabakan: fetch machine inventory or trigger repairs
c. Architectural Patterns & Principles
• Leader-election & shared state via distributed datastore (etcd).
• Command-pattern for CLI commands (Cobra in pkg/ckecli).
• Plugin/phase-based engine (op/ modules) enabling easy extension.
• Separation of concerns: CLI vs. API service vs. sidecar vs. op logic vs. persistence.
d. Key Technologies & Frameworks
• Go modules & standard library (SSH, HTTP, JSON).
• Cobra for CLI.
• gRPC or HTTP+JSON for internal API (HTTP in server/).
• etcd v3 client for coordination & storage.
• Vault API for PKI.
• Docker / CRI for node-level container orchestration.
• GraphQL client for Sabakan (sabakan/query.go).