How azure/fleet works
Project type and purpose
• KubeFleet is a Kubernetes “operator”-style orchestration system (a multi-cluster control plane) written in Go.
• It’s composed of two main agent components (hub-agent and member-agent), a token service (authtoken), a suite of custom controllers/reconcilers, a pluggable scheduler framework, webhooks, and CRD definitions.
• Its goal is to manage workloads and custom resources across multiple Kubernetes clusters (“treat each cluster as cattle”).
File structure
• Top-level “cmd/” folders define three binaries:
– authtoken (token issuance/refresh)
– hubagent (control-plane side)
– memberagent (data-plane side)
• “apis/” holds all CustomResourceDefinition (CRD) Go types (cluster, placement, works).
• “pkg/controllers/” contains operator reconcilers that watch K8s resources, compute placement decisions, propagate “Work” objects, handle rollouts, eviction, overrides, staged updates, etc.
• “pkg/scheduler/” implements a scheduler framework with filter/score plugins (affinity, taint-toleration, etc.).
• “pkg/webhook/” hosts validating admission-webhooks for CRDs.
• “charts/” and “config/” package Helm charts and raw YAML for deploying hub and member agents.
• “hack/” and “tools/” contain helper scripts for testing, Azure authentication, load testing.
• “test/” includes unit, integration, and e2e test suites.
README
• Confirms it’s a CNCF sandbox project for multi-cluster placement.
A. Kubernetes Control Plane (Hub Cluster)
1. Hub-agent Deployment (Docker container)
– Components inside:
• Authtoken client → requests tokens from external identity provider (Azure MSI or K8s Secret)
• Controllers (in pkg/controllers/*)
‑ ClusterInventory
‑ ClusterResourcePlacement
‑ Overrider, WorkGenerator, WorkPropagator
‑ Rollout, Updaterun
• Scheduler Framework (pkg/scheduler)
• Webhook server (pkg/webhook)
• Metrics & Admission endpoints
– Registers & serves CRDs (apis/placement, cluster API groups)
2. Kubernetes API Server
– CRDs installed via config/crd/bases or charts CRDs
– Hub-agent uses informers to watch Placement CRs
3. External Token Provider (Azure MSI or K8s secret)
– Authtoken → token_refresher, writer
B. Kubernetes Data Plane (Member Cluster)
1. Member-agent Deployment
– Watches “AppliedWorks” CRs (multicluster.x-k8s.io)
– Applies manifests to local cluster
– Reports status back to hub cluster via K8s API
2. Kubernetes API Server on member cluster
C. Multi-cluster communication
– Hub-agent uses K8s CustomResources in hub cluster (MemberCluster, InternalMemberCluster)
– Member-agent establishes connection to hub cluster API (via tokens)
– Work & status flow:
1. User creates ClusterResourcePlacement on hub
2. Hub controllers schedule clusters → create Work objects per member cluster
3. Member-agent syncer sees Work CRs, applies manifests to workloads
4. Member-agent updates AppliedWork status back to hub
5. Hub controllers observe AppliedWork status, update placement status
D. Admission & Validation
– Webhook server in hub-agent
• Validates incoming Placement or Override CRs
• Uses pkg/webhook handlers
E. Helm Charts & Deployment
– charts/hub-agent and charts/member-agent define Deployments, RBAC, ServiceAccounts, WebhookConfigurations
– config/crd manifests for installation
Architectural patterns & design principles
• Operator pattern (reconcile loops, informers, client-go)
• Pluggable scheduler framework (implements k8s scheduling plugin interface)
• Controller-as-code: one controller per resource type, clear separation of concerns
• Webhook admission for CRD validation
• Helm-based installation with CRDs first, followed by agents
• Separation of control plane (hub) vs. data plane (member)
Essential elements
• CRD installation as a separate step (via Helm or config)