How foundationdb/fdb-kubernetes-operator works
Project type and purpose
• This is a Kubernetes Operator (“fdb-kubernetes-operator”) written in Go.
• It manages FoundationDB clusters by watching custom resources (CRDs) on the Kubernetes API, reconciling their desired state, and orchestrating FoundationDB processes (pods, services, PVCs) accordingly.
File structure
• Top-level directories:
• api/v1beta2 – Go types for the FoundationDBCluster, Backup, and Restore CRDs
• controllers – Reconciliation logic for cluster, backup, restore, and helper controllers (addPods, updateConfigMap, etc.)
• config – Kustomize overlays (CRDs, deployments, RBAC, samples, tests)
• charts – Helm chart for installing CRDs and the operator
• fdbclient/pkg/fdbadminclient – Library to drive fdbcli / admin interactions
• foundationdb-kubernetes-sidecar – Init-container sidecar code and configs
• kubectl-fdb – A kubectl plugin for end-users to execute fdbcli commands via the operator
• e2e – End-to-end tests with fixtures and chaos-mesh failure injection
• scripts & sample-apps – Helper scripts and sample workloads
• Key config/build files: Dockerfile, Makefile, go.mod/sum, GitHub Actions workflows, goreleaser config.
README for additional insights
• Emphasizes the operator pattern: install CRDs, deploy operator, apply FoundationDBCluster resources, watch logs via kubectl, use kubectl fdb plugin.
• Mentions Helm chart (charts folder) and Kustomize-based testing overlays (config/tests).
• Highlights e2e tests with chaos-mesh injection to ensure resilience.
a. Main components (group into logical tiers):
1. Kubernetes Control Plane
– API Server (CRD definitions for FoundationDBCluster, Backup, Restore)
– etcd
2. FoundationDB Kubernetes Operator Deployment
– Manager Pod (controller-runtime)
• Watches CRDs → Enqueues reconciliation loops
• Controllers (cluster, backup, restore, plus helper controllers)
• Uses fdbclient (admin_client) to talk to FoundationDB processes
– Sidecar Pod (foundationdb-kubernetes-sidecar)
• Init container that injects monitors/config into FDB pods
– RBAC (ServiceAccount, Roles, RoleBindings)
– ConfigMaps & Secrets (TLS certs, blobstore credentials)
3. FoundationDB Cluster
– Storage Pods
– Transaction/Stateless Pods
– Services (headless, public)
– PersistentVolumeClaims
4. Client/CLI Interaction
– kubectl-fdb plugin → talks to API Server → operator spawns fdbcli exec
– Sample-apps → deploy workloads into the FDB cluster
5. CI & Testing
– e2e harness with kind/chaos-mesh → injects failures into the real Kubernetes cluster
– Kustomize test overlays under config/tests
b. Relationships & interactions:
– Operator ↔ Kubernetes API (watch CRDs, create/update/delete k8s resources)
– Operator → fdbclient library → FoundationDB processes (via gRPC/fdbcli)
– Sidecar init container ↔ Operator (via injected ConfigMap/Env)
– kubectl-fdb plugin ↔ API Server & Operator → fdbcli in running pods
– e2e tests ↔ chaos-mesh ↔ Kubernetes cluster hosting the operator & FDB cluster
c. Architectural patterns/design principles:
– Kubernetes Operator pattern (reconciliation loop, declarative CRDs)
– Controller-runtime framework for shared tooling (leader election, metrics, health checks)
– Sidecar pattern to bootstrap process config
– Library abstraction (fdbclient & pkg/fdbadminclient) for talking to FoundationDB
– GitOps-style config management (Kustomize under config/, Helm under charts/)
– Test-driven design: extensive unit tests in controllers, api, internal; E2E tests via config/tests + chaos-mesh
d. Key technologies/frameworks/libraries:
– Go & controller-runtime / Kubebuilder
– Kubernetes CRDs, RBAC, Kustomize, Helm
– FoundationDB client libraries & fdbcli
– Docker, Makefile, GitHub Actions (CI), goreleaser
– chaos-mesh for fault injection in tests
– kubectl plugin model for “kubectl fdb”
Detail without overthinking
• Logical component: CRDs, controllers, client libraries, sidecars, pods, external tools.