# mercari/spanner-autoscaler architecture

> Architecture diagram of the GitHub repository mercari/spanner-autoscaler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mercari/spanner-autoscaler
- Repository: https://github.com/mercari/spanner-autoscaler

## Overview

##### Project Type and Purpose
- This project is a Kubernetes Operator designed for scaling Google Cloud Spanner based on CPU utilization.
- It functions similarly to the Horizontal Pod Autoscaler, but instead targets Cloud Spanner compute capacity by watching custom resources (CRDs) like SpannerAutoscaler and SpannerAutoscaleSchedule.
- The operator integrates with GCP APIs (for Spanner changes and Cloud Monitoring) and works inside a Kubernetes cluster.

##### File Structure
- Top-level directories of interest include:
- api: Contains the CRD definitions for versioning (v1alpha1 and v1beta1) which specify the custom resources such as SpannerAutoscaler and SpannerAutoscaleSchedule.
- cmd: Houses the main entry point (main.go) for the operator.
- config: Provides Kubernetes manifests and kustomize files for deploying the operator (including CRDs, RBAC, webhooks, certmanager configuration, and Prometheus monitoring).
- internal: Contains the actual implementation and business logic:
- controller: Where the reconciliation loops for SpannerAutoscaler and SpannerAutoscaleSchedule live.
- scheduler: Implements scheduled scaling functionality.
- spanner: Contains integration logic for Cloud Spanner (and likely Cloud Monitoring).
- syncer and metrics: Handle state synchronization and performance metric reporting.
- The presence of Makefile, Dockerfile, GitHub workflows, and other build/deployment scripts indicates CI/CD support and containerized deployment.

##### Insights from the README
- The README explains the main workflow: monitoring CPU utilization with two main scenarios (scaling up when CPU is high and scaling down gradually to prevent disruptions).
- It highlights the use of CRDs to configure autoscaling and scheduled scaling.
- There is clear instruction on authentication, warning about GCP APIs, and the requirement for service account key management. This shows interaction with external services (GCP Spanner API and Cloud Monitoring API).

a. Main Components to Include:
- Kubernetes Cluster:
- Operator Deployment (controller-manager Pod)
- Webhook Server (for CRD validation and admission control configured via the “config/webhook” directory)
- RBAC and Service Accounts (ensuring proper permissions and authentication)
- Custom Resources:
- SpannerAutoscaler CRD and its corresponding object instances.
- SpannerAutoscaleSchedule CRD.
- Internal Controllers:
- The reconciliation loop inside the “internal/controller” for syncing desired state.
- The scheduler component (located in “internal/scheduler”) responsible for scheduled scaling tasks.
- The syncer component (located in “internal/syncer”) which ensures that the current state is updated to match desired capacity.
- The metrics module (for collecting and pushing performance metrics).
- External Dependencies:
- Google Cloud Spanner API: The target of autoscaling operations (scaling compute capacity up or down).
- Google Cloud Monitoring API: For querying CPU utilization metrics.
- GCP Authentication: Managed via service account secrets (or via Workload Identity as configured).

b. Relationships and Interactions:
- The operator controller watches for changes in SpannerAutoscaler and SpannerAutoscaleSchedule CRDs.
- When a change is detected, the controller triggers the reconciliation process:
- It calculates desired changes based on CPU utilization (gathered via Cloud Monitoring API).
- It calls the Cloud Spanner API to adjust compute capacity.
- The scheduler module handles “scheduled scaling” events, triggering temporary compute capacity adjustments.
- Metrics are gathered and aggregated by the metrics submodule, which can interact with Prometheus (see config/prometheus directory).
- Authentication components (configured through secrets and service accounts) mediate secure access to GCP resources.
- Webhooks validate custom resource definitions to ensure incoming configuration is correct.

c. Architectural Patterns and Design Principles:
- Operator Pattern: The system separates desired state (custom resource definitions) from current state (monitored by the controller), and reconciles the differences.
- Controller Reconciliation Loop: The main control loop ensures eventual consistency.
- Separation of Concerns: Different directories and internal modules (controller, scheduler, syncer, metrics) are designed to handle distinct parts of the workflow.
- Versioning: CRDs are versioned (v1alpha1 and v1beta1), which highlights evolving functionality and backward compatibility considerations.

d. Technologies and Frameworks:
- Written in Golang and likely built using frameworks such as Kubebuilder or Operator SDK.
- Utilizes Kubernetes CRDs to extend Kubernetes API functionality.
- Build and CI/CD are driven by Docker, Makefile, and GitHub workflows.
- Uses kustomize and kpt for Kubernetes resource configuration and deployment.
- Interacts with external Google Cloud services (Cloud Spanner, Cloud Monitoring).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Kubernetes Cluster"
        direction TB
        Main["Operator Main (main.go)"]:::entry
        Operator["Controller Manager Pod"]:::internal
        Webhook["Webhook Server (CRD Validation)"]:::internal
        subgraph "Custom Resources"
            direction TB
            CRD_A["SpannerAutoscaler CRD"]:::crd
            CRD_S["SpannerAutoscaleSchedule CRD"]:::crd
        end
        Main --> Operator
        Operator --> Webhook
        CRD_A -->|"triggers"| Operator
        CRD_S -->|"triggers"| Operator
    end

    subgraph "Operator Components"
        direction TB
        Controller_A["SpannerAutoscaler Controller"]:::internal
        Controller_S["SpannerAutoscaleSchedule Controller"]:::internal
        Scheduler["Scheduler"]:::internal
        Syncer["Syncer"]:::internal
        Metrics["Metrics Module"]:::internal
        Operator --> Controller_A
        Operator --> Controller_S
        Controller_A --> Scheduler
        Controller_A --> Syncer
        Controller_A --> Metrics
        Controller_S --> Scheduler
        Controller_S --> Syncer
        Controller_S --> Metrics
        Scheduler -->|"schedules"| Controller_A
    end

    subgraph "External Services"
        direction TB
        CloudMonitoring["GCP Monitoring API"]:::external
        CloudSpanner["GCP Spanner API"]:::external
        Prometheus["Prometheus Monitoring"]:::external
    end

    Auth["GCP Authentication (ServiceAccount/WorkloadIdentity)"]:::auth

    %% Data and control flows
    Controller_A -->|"queriesCPU"| CloudMonitoring
    Controller_S -->|"queriesCPU"| CloudMonitoring
    Controller_A -->|"scaleReq"| CloudSpanner
    Controller_S -->|"scaleReq"| CloudSpanner
    Metrics -->|"exportMetrics"| Prometheus
    Auth -.-> Controller_A
    Auth -.-> Controller_S

    %% Click Events
    click Main "https://github.com/mercari/spanner-autoscaler/blob/master/cmd/main.go"
    click CRD_A "https://github.com/mercari/spanner-autoscaler/blob/master/config/crd/bases/spanner.mercari.com_spannerautoscalers.yaml"
    click CRD_S "https://github.com/mercari/spanner-autoscaler/blob/master/config/crd/bases/spanner.mercari.com_spannerautoscaleschedules.yaml"
    click CRD_A "https://github.com/mercari/spanner-autoscaler/tree/master/api/v1alpha1"
    click CRD_S "https://github.com/mercari/spanner-autoscaler/tree/master/api/v1beta1"
    click Controller_A "https://github.com/mercari/spanner-autoscaler/blob/master/internal/controller/spannerautoscaler_controller.go"
    click Controller_S "https://github.com/mercari/spanner-autoscaler/blob/master/internal/controller/spannerautoscaleschedule_controller.go"
    click Scheduler "https://github.com/mercari/spanner-autoscaler/blob/master/internal/scheduler/scheduler.go"
    click Syncer "https://github.com/mercari/spanner-autoscaler/blob/master/internal/syncer/syncer.go"
    click Metrics "https://github.com/mercari/spanner-autoscaler/blob/master/internal/metrics/metrics.go"
    click Webhook "https://github.com/mercari/spanner-autoscaler/tree/master/config/webhook"
    click Prometheus "https://github.com/mercari/spanner-autoscaler/tree/master/config/prometheus"
    click CloudSpanner "https://github.com/mercari/spanner-autoscaler/blob/master/internal/spanner/spanner.go"

    %% Styling
    classDef internal fill:#bbf,stroke:#333,stroke-width:2px;
    classDef external fill:#fdb,stroke:#333,stroke-width:2px;
    classDef crd fill:#dfd,stroke:#333,stroke-width:2px;
    classDef auth fill:#fcc,stroke:#333,stroke-width:2px;
    classDef entry fill:#cfc,stroke:#333,stroke-width:2px;
```
