# googlecloudplatform/spanner-migration-tool architecture

> Architecture diagram of the GitHub repository googlecloudplatform/spanner-migration-tool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/spanner-migration-tool
- Repository: https://github.com/googlecloudplatform/spanner-migration-tool

## Overview

##### Project Type & Purpose
- This is a full-stack, multi-module migration tool written in Go, with both a CLI and a browser-based UI (Angular).
- Purpose: ingest schema/data from various sources (MySQL, PostgreSQL, Oracle, SQL Server, DynamoDB, CSV, Cassandra), analyze (“assessment”), convert to Spanner schema & data, support minimal-downtime streaming migration, and expose an interactive UI.

##### Top-Level Components
a. **User Interfaces**
- CLI (main.go + cmd/) → user drives migration via terminal commands.
- UI (ui/ folder) → Angular SPA served during development; production UI served via separate web server.
b. **Web API**
- webv2/ → Go HTTP server exposing REST endpoints that back the Angular UI.
c. **Core Engine**
- assessment/ → code analyzers, embedding generation, report generation.
- conversion/ → schema and data transformation logic.
- streaming/ & reverse_replication/ → minimal downtime and change-data-capture pipelines.
- dao/ & accessors/ → abstracted clients for Spanner, Dataflow, Datastream, Storage, ResourceManager.
- sources/ → connectors for each source type (MySQL, PostgreSQL, Oracle, SQL Server, DynamoDB, CSV, Cassandra).
d. **Shared Utilities**
- common/, internal/, logger/, expressions_api/, import_data/, profiles/, performance/ → helpers, logging, additional tools.

##### External Dependencies & Services
- Source Databases / Dump Files
(MySQL, PostgreSQL, Oracle, SQL Server, DynamoDB, Cassandra, CSV files)
- Google Cloud Services
- Cloud Spanner
- Cloud Dataflow
- Cloud Datastream
- Google Cloud Storage
- Cloud Resource Manager
- Pub/Sub notifications (for streaming)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Users
    User["User"]:::user

    %% Entry Points
    subgraph "User Interfaces"
        direction TB
        CLI["CLI"]:::cli
        UI["Angular UI"]:::frontend
    end

    User --> CLI
    User --> UI

    %% Backend
    subgraph "Backend"
        direction TB
        CLIApp["CLI Application"]:::backend
        WebAPI["webv2 HTTP Server"]:::backend
    end

    CLI --> CLIApp
    UI --> WebAPI

    %% Core Engine
    subgraph "Core Engine" 
        direction TB
        subgraph "Source Connector Layer"
            SRC["sources/"]:::core
            SRC_MY["sources/mysql/"]:::core
            SRC_PG["sources/postgres/"]:::core
            SRC_OR["sources/oracle/"]:::core
            SRC_SS["sources/sqlserver/"]:::core
            SRC_DDB["sources/dynamodb/"]:::core
            SRC_CSV["sources/csv/"]:::core
            SRC_CA["sources/cassandra/"]:::core
        end
        subgraph "Assessment Engine"
            ASSESS["assessment/"]:::core
        end
        subgraph "Conversion Engine"
            CONV["conversion/"]:::core
        end
        subgraph "Streaming & CDC"
            STREAM["streaming/"]:::core
            REVREP["reverse_replication/"]:::core
        end
        subgraph "DAO & Accessors"
            DAO["dao/"]:::adapter
            ACC["accessors/"]:::adapter
            ACC_CLI["accessors/clients/"]:::adapter
        end
        subgraph "Shared Utilities"
            COMMON["common/"]:::util
            INTERNAL["internal/"]:::util
            LOGGER["logger/"]:::util
            EXPR["expressions_api/"]:::util
            IMPORT["import_data/"]:::util
            PROFILES["profiles/"]:::util
            PERF["performance/"]:::util
        end
    end

    %% Connections to Core Engine
    CLIApp --> SRC
    WebAPI --> SRC

    SRC --> ASSESS
    SRC --> CONV

    CONV --> DAO
    ASSESS --> IMPORT
    ASSESS --> COMMON

    STREAM --> ACC
    REVREP --> ACC
    DAO --> ACC_CLI

    ACC -->|"Call GCP APIs"| CloudSpanner
    ACC -->|"Call GCP APIs"| CloudDataflow
    ACC -->|"Call GCP APIs"| CloudDatastream
    ACC -->|"Call GCP APIs"| CloudStorage
    ACC -->|"Call GCP APIs"| CloudResource

    STREAM --> PubSub
    STREAM --> CloudDatastream
    STREAM --> CloudSpanner

    %% Cloud Services
    subgraph "Google Cloud Services"
        direction TB
        CloudSpanner["Cloud Spanner"]:::cloud
        CloudDataflow["Cloud Dataflow"]:::cloud
        CloudDatastream["Cloud Datastream"]:::cloud
        CloudStorage["Cloud Storage"]:::cloud
        CloudResource["Cloud Resource Manager"]:::cloud
        PubSub["Pub/Sub"]:::cloud
    end

    %% Click Events
    click CLI "https://github.com/googlecloudplatform/spanner-migration-tool/blob/master/main.go"
    click CLI "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/cmd/"
    click UI "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/ui/"
    click WebAPI "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/webv2/"
    click ASSESS "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/assessment/"
    click CONV "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/conversion/"
    click STREAM "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/streaming/"
    click REVREP "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/reverse_replication/"
    click DAO "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/dao/"
    click ACC "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/accessors/"
    click ACC_CLI "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/accessors/clients/"
    click SRC "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/"
    click SRC_MY "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/mysql/"
    click SRC_PG "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/postgres/"
    click SRC_OR "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/oracle/"
    click SRC_SS "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/sqlserver/"
    click SRC_DDB "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/dynamodb/"
    click SRC_CSV "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/csv/"
    click SRC_CA "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/sources/cassandra/"
    click COMMON "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/common/"
    click INTERNAL "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/internal/"
    click LOGGER "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/logger/"
    click EXPR "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/expressions_api/"
    click IMPORT "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/import_data/"
    click PROFILES "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/profiles/"
    click PERF "https://github.com/googlecloudplatform/spanner-migration-tool/tree/master/performance/"

    %% Styles
    classDef user fill:#FFD966,stroke:#D79B00
    classDef frontend fill:#6FA8DC,stroke:#3D85C6
    classDef cli fill:#D9D2E9,stroke:#8E7CC3
    classDef backend fill:#B4A7D6,stroke:#674EA7
    classDef core fill:#FFD399,stroke:#E69138
    classDef adapter fill:#A4C2F4,stroke:#3C78D8
    classDef util fill:#EAD1DC,stroke:#C27BA0
    classDef cloud fill:#D5A6BD,stroke:#C27BA0
```
