How mercari/spanner-autoscaler works
Project Type and Purpose
• This project is a Kubernetes Operator designed for scaling Google Cloud Spanner based on CPU utilization.
• It functions similarly to the Horizontal Pod Autoscaler, but instead targets Cloud Spanner compute capacity by watching custom resources (CRDs) like SpannerAutoscaler and SpannerAutoscaleSchedule.
• The operator integrates with GCP APIs (for Spanner changes and Cloud Monitoring) and works inside a Kubernetes cluster.
File Structure
• Top-level directories of interest include:
• api: Contains the CRD definitions for versioning (v1alpha1 and v1beta1) which specify the custom resources such as SpannerAutoscaler and SpannerAutoscaleSchedule.
• cmd: Houses the main entry point (main.go) for the operator.
• config: Provides Kubernetes manifests and kustomize files for deploying the operator (including CRDs, RBAC, webhooks, certmanager configuration, and Prometheus monitoring).
• internal: Contains the actual implementation and business logic:
• controller: Where the reconciliation loops for SpannerAutoscaler and SpannerAutoscaleSchedule live.
• scheduler: Implements scheduled scaling functionality.
• spanner: Contains integration logic for Cloud Spanner (and likely Cloud Monitoring).
• syncer and metrics: Handle state synchronization and performance metric reporting.
• The presence of Makefile, Dockerfile, GitHub workflows, and other build/deployment scripts indicates CI/CD support and containerized deployment.
Insights from the README
• The README explains the main workflow: monitoring CPU utilization with two main scenarios (scaling up when CPU is high and scaling down gradually to prevent disruptions).
• It highlights the use of CRDs to configure autoscaling and scheduled scaling.
• There is clear instruction on authentication, warning about GCP APIs, and the requirement for service account key management. This shows interaction with external services (GCP Spanner API and Cloud Monitoring API).
a. Main Components to Include:
• Kubernetes Cluster:
• Operator Deployment (controller-manager Pod)
• Webhook Server (for CRD validation and admission control configured via the “config/webhook” directory)
• RBAC and Service Accounts (ensuring proper permissions and authentication)
• Custom Resources:
• SpannerAutoscaler CRD and its corresponding object instances.
• SpannerAutoscaleSchedule CRD.
• Internal Controllers:
• The reconciliation loop inside the “internal/controller” for syncing desired state.
• The scheduler component (located in “internal/scheduler”) responsible for scheduled scaling tasks.
• The syncer component (located in “internal/syncer”) which ensures that the current state is updated to match desired capacity.
• The metrics module (for collecting and pushing performance metrics).
• External Dependencies:
• Google Cloud Spanner API: The target of autoscaling operations (scaling compute capacity up or down).
• Google Cloud Monitoring API: For querying CPU utilization metrics.
• GCP Authentication: Managed via service account secrets (or via Workload Identity as configured).
b. Relationships and Interactions:
• The operator controller watches for changes in SpannerAutoscaler and SpannerAutoscaleSchedule CRDs.
• When a change is detected, the controller triggers the reconciliation process:
• It calculates desired changes based on CPU utilization (gathered via Cloud Monitoring API).
• It calls the Cloud Spanner API to adjust compute capacity.
• The scheduler module handles “scheduled scaling” events, triggering temporary compute capacity adjustments.
• Metrics are gathered and aggregated by the metrics submodule, which can interact with Prometheus (see config/prometheus directory).
• Authentication components (configured through secrets and service accounts) mediate secure access to GCP resources.
• Webhooks validate custom resource definitions to ensure incoming configuration is correct.
c. Architectural Patterns and Design Principles:
• Operator Pattern: The system separates desired state (custom resource definitions) from current state (monitored by the controller), and reconciles the differences.
• Controller Reconciliation Loop: The main control loop ensures eventual consistency.
• Separation of Concerns: Different directories and internal modules (controller, scheduler, syncer, metrics) are designed to handle distinct parts of the workflow.
• Versioning: CRDs are versioned (v1alpha1 and v1beta1), which highlights evolving functionality and backward compatibility considerations.
d. Technologies and Frameworks:
• Written in Golang and likely built using frameworks such as Kubebuilder or Operator SDK.
• Utilizes Kubernetes CRDs to extend Kubernetes API functionality.
• Build and CI/CD are driven by Docker, Makefile, and GitHub workflows.
• Uses kustomize and kpt for Kubernetes resource configuration and deployment.
• Interacts with external Google Cloud services (Cloud Spanner, Cloud Monitoring).