How google/dwh-migration-tools works
Below are detailed guidelines tailored for this project:
Main Components
• Metadata and Log Dumper Tool (found in the “dumper” directory): This Java-based application extracts DDL metadata and logs from various databases. It contains a rich set of connectors for different databases (Oracle, MySQL, Teradata, etc.) and for different storage backends (e.g., AWS S3, Google Cloud Storage, HDFS).
• Batch SQL Translation Client (found in the “client” directory): A command-line (and possibly Dockerizable) client developed in Python that submits translation jobs to BigQuery, handling macro expansion/unexpansion as required.
• DBSync Module (in “dbsync”): A set of Java submodules that appear to handle synchronization tasks – including byte source/sink adapters and various JDBC and cloud storage integration pieces.
• Shared libraries (found in “lib-common”, “lib-dumper-spi”, “lib-ext-bigquery”, “lib-ext-hive-metastore”): These provide common utilities, extension points, and SPI interfaces for the dumper and related functionalities.
• Build/Deployment tooling (Gradle scripts, buildSrc, Dockerfiles): Indicate the importance of Gradle as the build system and the availability of deployment support (e.g., Docker integration in the “client” folder).
Component Relationships
• The BigQuery Migration Service is the external consumer of metadata/log dumps and translation results; show it as an external (cloud) service.
• The Metadata and Log Dumper tool interacts with source databases via a set of connectors (using JDBC and custom protocols) and uses common libraries for tasks such as checksum generation, query execution, or file I/O.
• The Batch SQL Translation Client orchestrates SQL translation jobs and communicates with BigQuery Migration endpoints.
• The DBSync module is used by some tools (possibly the dumper) to perform low-level synchronization and supports different transport/storage abstractions (S3, GCS, HDFS).
• Shared libraries provide common functionality to both the dumper and dbsync modules.
Architectural Patterns and Design Principles
• Modular design: The project is clearly structured as a set of independent sub-projects that build Java libraries and applications, alongside a separate Python-based client.
• Connector/Plugin Pattern: In the dumper, numerous subpackages under “connector” (with support for Oracle, Teradata, BigQuery, etc.) indicate a pluggable architecture that allows adding new data sources or sinks.
• Separation of concerns: The separation between core application logic (in “dumper/app”, “dbsync”) and extension points (in “lib-dumper-spi”, “lib-ext-*”) is critical.
• Cloud integration: The README emphasizes compatibility with the BigQuery Migration Service, indicating that one end of the system interacts with cloud APIs.
Technologies, Frameworks, and Libraries to Note
• Java (JDK 8 or higher) is used for the main applications (dumper, dbsync).
• Gradle is used both as a build and dependency management tool.
• Python is used for the SQL translation client.
• Various third-party JDBC drivers (not included in the repository but expected at runtime) for connecting to data source databases.
• Docker (for packaging and potentially deployment in the “client” directory).
Essential Elements
• Components: “Metadata & Log Dumper”, “Batch SQL Translation Client”, “DBSync Infrastructure”, “Shared Libraries/Plugins”, and external services like “BigQuery Migration Service”.
• Core application modules (Java applications, Python client).
• Supporting libraries (common, SPI, extensions).
• External systems (databases, BigQuery, cloud storage).
• A multi-layered approach:
• Top layer: External Interfaces (user interfaces, command line, cloud APIs).
• Middle layer: Application logic (dumper, client, dbsync).
• Lower layer: Shared libraries and connectors.
• Side note: Build and deployment tools can be represented as an overlay that facilitates the entire build process.