How data-integrations/google-cloud works
This project is a collection of CDAP (Cask Data Application Platform) plugins that enable integration with various Google Cloud Platform (GCP) services. The repository implements plugins to support different types of connectors and actions – including Cloud BigQuery (for source, sink, and multi-sink operations), Google Cloud Storage (source, sink, create, delete and move actions), Cloud Pub/Sub (source and sink), Cloud Spanner (source and sink), Cloud Speech API and Google Cloud Datastore (source and sink). The purpose is to extend CDAP’s functionality by offering standardized, reusable GCP integrations.
Main Components
• CDAP Plugin Framework: At the core, these plugins are designed to integrate with CDAP. Represent the CDAP platform (or the runtime environment) as a central component that loads, manages, and executes these plugins.
• BigQuery Plugins: Include connectors, actions (e.g., BigQueryArgumentSetter, BigQueryExecute), and the SQL engine components (which handle query construction and execution).
• GCS Plugins: Represent sources, sinks, and specialized actions (e.g., GCSBucketCreate, GCSBucketDelete, GCSDoneFileMarker, GCSCopy, GCSMove).
• Pub/Sub Plugins: Include components for both source/sink and any subscriber logic.
• Spanner Plugins: Indicate source and sink modules alongside helper utilities for mutation transformation.
• Common / Shared Libraries: There are common utility packages and error handling support within “io.cdap.plugin/gcp/common” that should be shown as shared modules used across the plugins.
• External Systems: List the corresponding target GCP services (BigQuery, Cloud Storage, Pub/Sub, Spanner, Datastore, Speech API) as external systems that the plugins interact with.
Relationships and Interactions
• If needed, show the interactions between sub-components within a plugin. For example, in the BigQuery module, show the relation between the connector, the SQL engine (including builder classes like BigQuerySelectSQLBuilder), and the record converters.
Design Patterns and Architectural Principles
• Modularity and separation of responsibilities: each plugin is self-contained (with its own actions, connectors, and utilities) while conforming to the CDAP plugin model.
• How the project uses common error handling, configuration, and utility layers that are reused across different plugins.