# google/dwh-migration-tools architecture

> Architecture diagram of the GitHub repository google/dwh-migration-tools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google/dwh-migration-tools
- Repository: https://github.com/google/dwh-migration-tools

## Overview

Below are detailed guidelines tailored for this project:

##### Main Components
- Metadata and Log Dumper Tool (found in the “dumper” directory): This Java-based application extracts DDL metadata and logs from various databases. It contains a rich set of connectors for different databases (Oracle, MySQL, Teradata, etc.) and for different storage backends (e.g., AWS S3, Google Cloud Storage, HDFS).
- Batch SQL Translation Client (found in the “client” directory): A command-line (and possibly Dockerizable) client developed in Python that submits translation jobs to BigQuery, handling macro expansion/unexpansion as required.
- DBSync Module (in “dbsync”): A set of Java submodules that appear to handle synchronization tasks – including byte source/sink adapters and various JDBC and cloud storage integration pieces.
- Shared libraries (found in “lib-common”, “lib-dumper-spi”, “lib-ext-bigquery”, “lib-ext-hive-metastore”): These provide common utilities, extension points, and SPI interfaces for the dumper and related functionalities.
- Build/Deployment tooling (Gradle scripts, buildSrc, Dockerfiles): Indicate the importance of Gradle as the build system and the availability of deployment support (e.g., Docker integration in the “client” folder).

##### Component Relationships
- The BigQuery Migration Service is the external consumer of metadata/log dumps and translation results; show it as an external (cloud) service.
- The Metadata and Log Dumper tool interacts with source databases via a set of connectors (using JDBC and custom protocols) and uses common libraries for tasks such as checksum generation, query execution, or file I/O.
- The Batch SQL Translation Client orchestrates SQL translation jobs and communicates with BigQuery Migration endpoints.
- The DBSync module is used by some tools (possibly the dumper) to perform low-level synchronization and supports different transport/storage abstractions (S3, GCS, HDFS).
- Shared libraries provide common functionality to both the dumper and dbsync modules.

##### Architectural Patterns and Design Principles
- Modular design: The project is clearly structured as a set of independent sub-projects that build Java libraries and applications, alongside a separate Python-based client.
- Connector/Plugin Pattern: In the dumper, numerous subpackages under “connector” (with support for Oracle, Teradata, BigQuery, etc.) indicate a pluggable architecture that allows adding new data sources or sinks.
- Separation of concerns: The separation between core application logic (in “dumper/app”, “dbsync”) and extension points (in “lib-dumper-spi”, “lib-ext-*”) is critical.
- Cloud integration: The README emphasizes compatibility with the BigQuery Migration Service, indicating that one end of the system interacts with cloud APIs.

##### Technologies, Frameworks, and Libraries to Note
- Java (JDK 8 or higher) is used for the main applications (dumper, dbsync).
- Gradle is used both as a build and dependency management tool.
- Python is used for the SQL translation client.
- Various third-party JDBC drivers (not included in the repository but expected at runtime) for connecting to data source databases.
- Docker (for packaging and potentially deployment in the “client” directory).

##### Essential Elements
- Components: “Metadata & Log Dumper”, “Batch SQL Translation Client”, “DBSync Infrastructure”, “Shared Libraries/Plugins”, and external services like “BigQuery Migration Service”.
- Core application modules (Java applications, Python client).
- Supporting libraries (common, SPI, extensions).
- External systems (databases, BigQuery, cloud storage).

- A multi-layered approach:
- Top layer: External Interfaces (user interfaces, command line, cloud APIs).
- Middle layer: Application logic (dumper, client, dbsync).
- Lower layer: Shared libraries and connectors.
- Side note: Build and deployment tools can be represented as an overlay that facilitates the entire build process.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Systems
    subgraph "External Systems"
        SD["Source Databases"]:::external
        BQ["BigQuery Migration Service"]:::external
    end

    %% Application Layer
    subgraph "Application Layer"
        Dumper["Metadata & Log Dumper (Java)"]:::java
        Client["Batch SQL Translation Client (Python)"]:::python
        DBSync["DBSync Module (Java)"]:::java
    end

    %% Shared Libraries & Plugins
    subgraph "Shared Libraries & Plugins"
        Common["Common Utilities"]:::shared
        SPI["Dumper SPI"]:::shared
        BQExt["BigQuery Extensions"]:::shared
        HiveExt["Hive Metastore Extensions"]:::shared
    end

    %% Build & Deployment Tools
    subgraph "Build & Deployment Tools"
        Gradle["Gradle Scripts (build.gradle, buildSrc, gradle)"]:::build
        Docker["Dockerfile (client/Dockerfile)"]:::build
    end

    %% Relationships
    SD -->|"extracts metadata & logs"| Dumper
    Dumper -->|"sends dumps"| BQ
    Client -->|"submits SQL translation jobs"| BQ

    Dumper -->|"uses"| Common
    Dumper -->|"uses"| SPI
    Dumper -->|"uses"| BQExt
    Dumper -->|"uses"| HiveExt

    DBSync -->|"uses"| Common
    DBSync -->|"uses"| SPI

    Gradle -->|"builds"| Dumper
    Gradle -->|"builds"| DBSync
    Gradle -->|"builds"| Client
    Docker -->|"packages"| Client

    %% Click Events
    click Dumper "https://github.com/google/dwh-migration-tools/tree/main/dumper/app"
    click Client "https://github.com/google/dwh-migration-tools/tree/main/client"
    click DBSync "https://github.com/google/dwh-migration-tools/tree/main/dbsync"
    click Common "https://github.com/google/dwh-migration-tools/tree/main/dumper/lib-common"
    click SPI "https://github.com/google/dwh-migration-tools/tree/main/dumper/lib-dumper-spi"
    click BQExt "https://github.com/google/dwh-migration-tools/tree/main/dumper/lib-ext-bigquery"
    click HiveExt "https://github.com/google/dwh-migration-tools/tree/main/dumper/lib-ext-hive-metastore"
    click Gradle "https://github.com/google/dwh-migration-tools/blob/main/build.gradle"
    click Docker "https://github.com/google/dwh-migration-tools/tree/main/client/Dockerfile"

    %% Styles
    classDef java fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef python fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef external fill:#fffacd,stroke:#333,stroke-width:2px;
    classDef shared fill:#ffa07a,stroke:#333,stroke-width:2px;
    classDef build fill:#dda0dd,stroke:#333,stroke-width:2px;
```
