# azure/fleet architecture

> Architecture diagram of the GitHub repository azure/fleet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure/fleet
- Repository: https://github.com/azure/fleet

## Overview

##### Project type and purpose
- KubeFleet is a Kubernetes “operator”-style orchestration system (a multi-cluster control plane) written in Go.
- It’s composed of two main agent components (hub-agent and member-agent), a token service (authtoken), a suite of custom controllers/reconcilers, a pluggable scheduler framework, webhooks, and CRD definitions.
- Its goal is to manage workloads and custom resources across multiple Kubernetes clusters (“treat each cluster as cattle”).

##### File structure
- Top-level “cmd/” folders define three binaries:
– authtoken (token issuance/refresh)
– hubagent (control-plane side)
– memberagent (data-plane side)
- “apis/” holds all CustomResourceDefinition (CRD) Go types (cluster, placement, works).
- “pkg/controllers/” contains operator reconcilers that watch K8s resources, compute placement decisions, propagate “Work” objects, handle rollouts, eviction, overrides, staged updates, etc.
- “pkg/scheduler/” implements a scheduler framework with filter/score plugins (affinity, taint-toleration, etc.).
- “pkg/webhook/” hosts validating admission-webhooks for CRDs.
- “charts/” and “config/” package Helm charts and raw YAML for deploying hub and member agents.
- “hack/” and “tools/” contain helper scripts for testing, Azure authentication, load testing.
- “test/” includes unit, integration, and e2e test suites.

##### README
- Confirms it’s a CNCF sandbox project for multi-cluster placement.

A. Kubernetes Control Plane (Hub Cluster)
1. Hub-agent Deployment (Docker container)
– Components inside:
- Authtoken client → requests tokens from external identity provider (Azure MSI or K8s Secret)
- Controllers (in pkg/controllers/*)
‑ ClusterInventory
‑ ClusterResourcePlacement
‑ Overrider, WorkGenerator, WorkPropagator
‑ Rollout, Updaterun
- Scheduler Framework (pkg/scheduler)
- Webhook server (pkg/webhook)
- Metrics & Admission endpoints
– Registers & serves CRDs (apis/placement, cluster API groups)
2. Kubernetes API Server
– CRDs installed via config/crd/bases or charts CRDs
– Hub-agent uses informers to watch Placement CRs
3. External Token Provider (Azure MSI or K8s secret)
– Authtoken → token_refresher, writer

B. Kubernetes Data Plane (Member Cluster)
1. Member-agent Deployment
– Watches “AppliedWorks” CRs (multicluster.x-k8s.io)
– Applies manifests to local cluster
– Reports status back to hub cluster via K8s API
2. Kubernetes API Server on member cluster

C. Multi-cluster communication
– Hub-agent uses K8s CustomResources in hub cluster (MemberCluster, InternalMemberCluster)
– Member-agent establishes connection to hub cluster API (via tokens)
– Work & status flow:
1. User creates ClusterResourcePlacement on hub
2. Hub controllers schedule clusters → create Work objects per member cluster
3. Member-agent syncer sees Work CRs, applies manifests to workloads
4. Member-agent updates AppliedWork status back to hub
5. Hub controllers observe AppliedWork status, update placement status

D. Admission & Validation
– Webhook server in hub-agent
- Validates incoming Placement or Override CRs
- Uses pkg/webhook handlers

E. Helm Charts & Deployment
– charts/hub-agent and charts/member-agent define Deployments, RBAC, ServiceAccounts, WebhookConfigurations
– config/crd manifests for installation

##### Architectural patterns & design principles
- Operator pattern (reconcile loops, informers, client-go)
- Pluggable scheduler framework (implements k8s scheduling plugin interface)
- Controller-as-code: one controller per resource type, clear separation of concerns
- Webhook admission for CRD validation
- Helm-based installation with CRDs first, followed by agents
- Separation of control plane (hub) vs. data plane (member)

##### Essential elements
- CRD installation as a separate step (via Helm or config)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Hub Cluster
    subgraph "Hub Cluster"
        direction TB
        HA_API["Hub API Server"]:::control
        subgraph "Hub-Agent Pod"
            direction TB
            Controllers["Controllers\n(pkg/controllers)"]:::control
            Scheduler["Scheduler Framework\n(pkg/scheduler)"]:::control
            Webhook["Admission Webhook Server\n(pkg/webhook)"]:::control
            Authtoken["Authtoken Service\n(cmd/authtoken)"]:::control
        end
        HA_API -->|"Watch CRDs"| Controllers
        Controllers -->|"Schedule & Create Work CRs"| HA_API
        Controllers -->|"Uses"| Scheduler
        HA_API -.->|"Validation Requests"| Webhook
        Authtoken -->|"Request Token"| IdentityProvider
        IdentityProvider -->|"Return Token"| Authtoken
    end

    %% Member Cluster
    subgraph "Member Cluster"
        direction TB
        MA_API["Member API Server"]:::control
        subgraph "Member-Agent Pod"
            direction TB
            Syncer["AppliedWork Syncer & Applier\n(pkg/controllers/work)"]:::dataplane
        end
        MA_API -->|"Watch Work CRs"| Syncer
        Syncer -->|"Apply Work Manifests"| MA_API
        Syncer -->|"Update Status"| HA_API
    end

    %% External Services & Config
    IdentityProvider["External Token Provider\n(Azure MSI / K8s Secret)"]:::external
    CRDInstall["CRD Installation\n(config/crd/bases)"]:::config
    HelmCharts["Helm Charts\n(charts/hub-agent, charts/member-agent)"]:::config

    %% Installation Flow
    CRDInstall -->|"CRDs Installed"| HA_API
    HelmCharts -->|"Deploy Agents & Webhooks"| HA_API
    HelmCharts -->|"Deploy Agents & Syncer"| MA_API

    %% Click Events
    click Authtoken "https://github.com/azure/fleet/blob/main/cmd/authtoken/main.go"
    click Authtoken "https://github.com/azure/fleet/tree/main/pkg/authtoken/"
    click Controllers "https://github.com/azure/fleet/tree/main/pkg/controllers/"
    click Scheduler "https://github.com/azure/fleet/tree/main/pkg/scheduler/framework/"
    click Webhook "https://github.com/azure/fleet/tree/main/pkg/webhook/"
    click Syncer "https://github.com/azure/fleet/blob/main/pkg/controllers/work/applied_work_syncer.go"
    click Syncer "https://github.com/azure/fleet/tree/main/pkg/controllers/workapplier/"
    click CRDInstall "https://github.com/azure/fleet/tree/main/config/crd/bases/"
    click HelmCharts "https://github.com/azure/fleet/tree/main/charts/hub-agent/"
    click HelmCharts "https://github.com/azure/fleet/tree/main/charts/member-agent/"
    click HA_API "https://github.com/azure/fleet/blob/main/cmd/hubagent/main.go"
    click MA_API "https://github.com/azure/fleet/blob/main/cmd/memberagent/main.go"

    %% Styles
    classDef control fill:#E3F2FD,stroke:#1E88E5,color:#0D47A1;
    classDef dataplane fill:#E8F5E9,stroke:#43A047,color:#1B5E20;
    classDef external fill:#FFF9C4,stroke:#FBC02D,color:#F57F17;
    classDef config fill:#F3E5F5,stroke:#8E24AA,color:#4A148C;
```
