# data-integrations/google-cloud architecture

> Architecture diagram of the GitHub repository data-integrations/google-cloud, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/data-integrations/google-cloud
- Repository: https://github.com/data-integrations/google-cloud

## Overview

This project is a collection of CDAP (Cask Data Application Platform) plugins that enable integration with various Google Cloud Platform (GCP) services. The repository implements plugins to support different types of connectors and actions – including Cloud BigQuery (for source, sink, and multi-sink operations), Google Cloud Storage (source, sink, create, delete and move actions), Cloud Pub/Sub (source and sink), Cloud Spanner (source and sink), Cloud Speech API and Google Cloud Datastore (source and sink). The purpose is to extend CDAP’s functionality by offering standardized, reusable GCP integrations.

##### Main Components
- CDAP Plugin Framework: At the core, these plugins are designed to integrate with CDAP. Represent the CDAP platform (or the runtime environment) as a central component that loads, manages, and executes these plugins.
- BigQuery Plugins: Include connectors, actions (e.g., BigQueryArgumentSetter, BigQueryExecute), and the SQL engine components (which handle query construction and execution).
- GCS Plugins: Represent sources, sinks, and specialized actions (e.g., GCSBucketCreate, GCSBucketDelete, GCSDoneFileMarker, GCSCopy, GCSMove).
- Pub/Sub Plugins: Include components for both source/sink and any subscriber logic.
- Spanner Plugins: Indicate source and sink modules alongside helper utilities for mutation transformation.
- Common / Shared Libraries: There are common utility packages and error handling support within “io.cdap.plugin/gcp/common” that should be shown as shared modules used across the plugins.
- External Systems: List the corresponding target GCP services (BigQuery, Cloud Storage, Pub/Sub, Spanner, Datastore, Speech API) as external systems that the plugins interact with.

##### Relationships and Interactions
- If needed, show the interactions between sub-components within a plugin. For example, in the BigQuery module, show the relation between the connector, the SQL engine (including builder classes like BigQuerySelectSQLBuilder), and the record converters.

##### Design Patterns and Architectural Principles
- Modularity and separation of responsibilities: each plugin is self-contained (with its own actions, connectors, and utilities) while conforming to the CDAP plugin model.
- How the project uses common error handling, configuration, and utility layers that are reused across different plugins.

## Diagram (Mermaid)

```mermaid
flowchart TD
    CP["CDAP Platform/Plugin Framework"]:::platform

    subgraph "Plugin Modules"
        subgraph "BigQuery Plugins"
            BQActions["BigQuery Actions"]:::plugin
            BQConnector["BigQuery Connector"]:::plugin
            BQSink["BigQuery Sink"]:::plugin
            BQSource["BigQuery Source"]:::plugin
            BQSQL["BigQuery SQL Engine"]:::plugin
        end

        subgraph "Google Cloud Storage (GCS) Plugins"
            GCSActions["GCS Actions"]:::plugin
            GCSConnector["GCS Connector"]:::plugin
            GCSSink["GCS Sink"]:::plugin
            GCSSource["GCS Source"]:::plugin
        end

        subgraph "Pub/Sub Plugins"
            PubSubSource["Pub/Sub Source"]:::plugin
            PubSubSink["Pub/Sub Sink"]:::plugin
        end

        subgraph "Spanner Plugins"
            SpannerConnector["Spanner Connector"]:::plugin
            SpannerSink["Spanner Sink"]:::plugin
            SpannerSource["Spanner Source"]:::plugin
        end

        subgraph "Datastore Plugins"
            DatastoreSink["Datastore Sink"]:::plugin
            DatastoreSource["Datastore Source"]:::plugin
        end

        subgraph "Speech API Plugin"
            SpeechTransform["Speech To Text Transform"]:::plugin
        end
    end

    Common["Common Utilities"]:::common

    %% External Systems
    BQExternal(("GCP BigQuery")):::external
    StorageExternal(("GCP Cloud Storage")):::external
    PubSubExternal(("GCP Pub/Sub")):::external
    SpannerExternal(("GCP Cloud Spanner")):::external
    DatastoreExternal(("GCP Datastore")):::external
    SpeechAPIExternal(("GCP Speech API")):::external

    %% Relationships: CDAP loads each plugin
    CP -->|"loads"| BQActions
    CP -->|"loads"| BQConnector
    CP -->|"loads"| BQSink
    CP -->|"loads"| BQSource
    CP -->|"loads"| BQSQL
    CP -->|"loads"| GCSActions
    CP -->|"loads"| GCSConnector
    CP -->|"loads"| GCSSink
    CP -->|"loads"| GCSSource
    CP -->|"loads"| PubSubSource
    CP -->|"loads"| PubSubSink
    CP -->|"loads"| SpannerConnector
    CP -->|"loads"| SpannerSink
    CP -->|"loads"| SpannerSource
    CP -->|"loads"| DatastoreSink
    CP -->|"loads"| DatastoreSource
    CP -->|"loads"| SpeechTransform

    %% Plugins use common utilities
    BQActions -->|"uses"| Common
    BQConnector -->|"uses"| Common
    BQSink -->|"uses"| Common
    BQSource -->|"uses"| Common
    BQSQL -->|"uses"| Common

    GCSActions -->|"uses"| Common
    GCSConnector -->|"uses"| Common
    GCSSink -->|"uses"| Common
    GCSSource -->|"uses"| Common

    PubSubSource -->|"uses"| Common
    PubSubSink -->|"uses"| Common

    SpannerConnector -->|"uses"| Common
    SpannerSink -->|"uses"| Common
    SpannerSource -->|"uses"| Common

    DatastoreSink -->|"uses"| Common
    DatastoreSource -->|"uses"| Common

    SpeechTransform -->|"uses"| Common

    %% Plugin interactions with external GCP services
    BQConnector -->|"interacts"| BQExternal
    BQSink -->|"writesTo"| BQExternal
    BQSource -->|"readsFrom"| BQExternal

    GCSConnector -->|"interacts"| StorageExternal
    GCSSink -->|"writesTo"| StorageExternal
    GCSSource -->|"readsFrom"| StorageExternal

    PubSubSource -->|"subscribesTo"| PubSubExternal
    PubSubSink -->|"publishesTo"| PubSubExternal

    SpannerConnector -->|"connectsTo"| SpannerExternal
    SpannerSink -->|"writesTo"| SpannerExternal
    SpannerSource -->|"readsFrom"| SpannerExternal

    DatastoreSink -->|"writesTo"| DatastoreExternal
    DatastoreSource -->|"readsFrom"| DatastoreExternal

    SpeechTransform -->|"transformsSpeech"| SpeechAPIExternal

    %% Click Events
    click CP "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp"
    click BQActions "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/bigquery/action"
    click BQConnector "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/bigquery/connector"
    click BQSink "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/bigquery/sink"
    click BQSource "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/bigquery/source"
    click BQSQL "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/bigquery/sqlengine"

    click GCSActions "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/gcs/actions"
    click GCSConnector "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/gcs/connector"
    click GCSSink "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/gcs/sink"
    click GCSSource "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/gcs/source"

    click PubSubSource "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/publisher/source"
    click PubSubSink "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/publisher"

    click SpannerConnector "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/spanner/connector"
    click SpannerSink "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/spanner/sink"
    click SpannerSource "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/spanner/source"

    click DatastoreSink "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/datastore/sink"
    click DatastoreSource "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/datastore/source"

    click SpeechTransform "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/speech"

    click Common "https://github.com/data-integrations/google-cloud/tree/develop/src/main/java/io/cdap/plugin/gcp/common"

    %% Styles
    classDef platform fill:#b3cde0,stroke:#333,stroke-width:2px;
    classDef plugin fill:#ccebc5,stroke:#333,stroke-width:2px;
    classDef external fill:#decbe4,stroke:#333,stroke-width:2px,stroke-dasharray:5,5;
    classDef common fill:#fbb4ae,stroke:#333,stroke-width:2px;
```
