# googlecloudplatform/serverless-spark-workshop architecture

> Architecture diagram of the GitHub repository googlecloudplatform/serverless-spark-workshop, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/googlecloudplatform/serverless-spark-workshop
- Repository: https://github.com/googlecloudplatform/serverless-spark-workshop

## Overview

##### Project Type and High‐Level Purpose
- Monorepo of Spark-based, Serverless pipelines and ML/analytics notebooks on GCP.
- Each subfolder under the root (e.g., `cell-tower-anomaly-detection-dbt`, `timeseries_forecasting`, `s8s-spark-mlops`, etc.) is a standalone solution accelerator demonstrating a common Spark use case (batch ETL, streaming, interactive ML, etc.).

##### File Structure Patterns
- Each accelerator contains:
– `00-scripts` or `02-scripts`: Spark job entrypoints (PySpark, Scala, SQL scripts, or notebooks).
– `01-datasets`: sample data or pointers to GCS/BigQuery tables.
– `02-dependencies` (if present): Dockerfiles or extra libs for containerized jobs.
– `instructions` folder: GCP setup steps (Terraform, Composer, BigQuery dataset creation).
- Root also holds Terraform configs for shared prerequisites (`s8s-prerequisites/`) and a global README listing all accelerators.

##### README Insights
- Common architecture:
– Infrastructure provisioning via Terraform → GCS buckets, BigQuery datasets, Dataproc Serverless endpoint, Dataproc Metastore, Vertex AI resources, Cloud Composer, Cloud Functions, Cloud Scheduler.
– Data ingestion: upload to GCS or directly to BigQuery/BigLake.
– Orchestration: Cloud Composer (Airflow DAGs), Cloud Scheduler + Cloud Functions, or manual gcloud commands.
– Compute: Dataproc Serverless Spark Batches or interactive Serverless Spark sessions within Vertex AI notebooks.
– Storage & catalog: BigQuery tables or BigLake external tables (Parquet/CSV) registered via Dataproc Metastore.
– Downstream uses: analytics dashboards, ML model training & serving in Vertex AI.

##### Main Components & Boundaries
a. User / Developer Environment
– Local IDE or Cloud Shell
– Vertex AI Workbench notebooks
– gcloud CLI
b. Infrastructure as Code (IaC)
– Terraform → provision GCS, BigQuery, Dataproc Serverless, Composer, Metastore, Vertex AI.
c. Orchestration Layer
– Cloud Composer (Airflow DAGs)
– Cloud Scheduler → triggers Cloud Functions → invokes Dataproc Serverless jobs
– Vertex AI Pipelines (for ML pipelines)
d. Compute Layer
– Dataproc Serverless Spark Batches (batch ETL, streaming, SQL, ML jobs)
– Interactive Serverless Spark on Vertex AI notebooks
e. Storage Layer
– Google Cloud Storage (raw data, Parquet/CSV staging)
– BigQuery / BigLake external tables (via Dataproc Metastore)
– Dataproc Metastore (Hive-compatible catalog)
f. ML & Serving
– Vertex AI Training & Endpoints (built‐in in some accelerators)
– Cloud Functions (e.g., for event‐driven triggers)

##### Relationships & Data Flows
– Data upload → GCS
– GCS → Dataproc Serverless (Spark reads staging data)
– Spark writes → BigQuery or back to GCS (transformed/parquet)
– BigQuery → BI or ML pipelines
– Orchestrator → triggers Compute
– IaC (Terraform) → provisions all resources
- For streaming solutions, show Pub/Sub or JSON files directory feeding Spark Structured Streaming.

##### Architectural Patterns
- ETL Batch Pattern: GCS → Spark Batch → BigQuery
- Streaming Pattern: Pub/Sub / GCS streaming source → Spark Structured Streaming → BigQuery
- Interactive ML Pattern: Vertex AI Notebook → Serverless Spark session → BigQuery → Vertex AI Training
- SQL–only Pattern: Spark SQL via Dataproc Serverless + Metastore → BigQuery

##### Technologies & Frameworks
- Dataproc Serverless Spark
- BigQuery & BigLake (external tables)
- Dataproc Metastore
- Cloud Composer (Airflow)
- Terraform for IaC
- Vertex AI Workbench & Pipelines
- Cloud Functions & Scheduler
- dbt (only in `cell-tower-anomaly-detection-dbt`)

##### Detail Is Key—Don’t Overthink
- Every distinct component you see in the file tree and README.
- Even if two accelerators use the same service, show them separately if they run in parallel or via different orchestrators.
- Always annotate any terraform modules or setup scripts as a “Provisioning” component.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Environment
    subgraph "User Environment"
        IDE["Local IDE / Cloud Shell"]:::code
        CLI["gcloud CLI"]:::code
        Notebook["Vertex AI Workbench"]:::code
    end

    %% Infrastructure as Code
    subgraph "Infrastructure as Code"
        TF_Shared["Terraform: Shared Prerequisites"]:::code
        TF_MLOps_Shared["Terraform: MLOps Shared Env"]:::code
        TF_MLOps["Terraform: MLOps Env"]:::code
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Composer["Cloud Composer (Airflow)"]:::gcp
        MLOps_DAG["Airflow DAG: Serverless Spark MLOps"]:::code
        CellTower_DAG["Airflow DAG: Cell Tower Anomaly Detection"]:::code
        Malware_DAG["Airflow DAG: Malware Detection"]:::code
        Retail_DAG["Airflow DAG: Retail Store Analytics"]:::code
        Scheduler["Cloud Scheduler"]:::gcp
        Functions["Cloud Functions Trigger"]:::gcp
        VAI_Pipeline["Vertex AI Pipeline"]:::gcp
    end

    %% Compute Layer
    subgraph "Compute Layer"
        SparkBatch["Dataproc Serverless Spark Batch"]:::gcp
        CommonUtils["PySpark: common_utils.py"]:::code
        ModelTrain["PySpark: model_training.py"]:::code
        BatchScore["PySpark: batch_scoring.py"]:::code
        ScalaJob["Scala Spark Job: covid-economic-impact-assessment.scala"]:::code
        PythonETL["PySpark Job: covid_economic_impact_etl.py"]:::code
        NotebookETL["Notebook ETL: covid_economic_impact_etl.ipynb"]:::code
        NotebookML["Notebook ML: customer_churn_model_building.ipynb"]:::code
        StreamingJob["Spark Streaming Job: spark_streaming_invoice.py"]:::code
    end

    %% Storage Layer
    subgraph "Storage Layer"
        GCS["Google Cloud Storage"]:::data
        BQ["BigQuery / BigLake"]:::data
        DS_TS["Dataset: timeseries train.csv"]:::data
        DS_Churn["Dataset: customer_churn_train_data.csv"]:::data
        DS_Parquet["Dataset: customers_raw_data/*.parquet"]:::data
        BQ_WC_Script["BigQuery Example Script"]:::code
    end

    %% Metadata Layer
    subgraph "Metadata Layer"
        Metastore["Dataproc Metastore"]:::gcp
        Metastore_SQL["Schema SQL: retail_analytics_schema_definition_sparksql.sql"]:::code
    end

    %% Dependencies & Templates
    subgraph "Dependencies & Templates"
        Docker_SM["Dockerfile: Social Media Analytics"]:::deps
        GraphFrames["GraphFrames Library"]:::deps
        VAI_Template["Vertex AI Pipeline Template"]:::template
    end

    %% Relationships
    IDE -->|provisions via Terraform| TF_Shared
    IDE --> TF_MLOps_Shared
    IDE --> TF_MLOps
    CLI --> TF_Shared
    CLI --> TF_MLOps_Shared
    CLI --> TF_MLOps

    TF_Shared --> GCS
    TF_Shared --> BQ
    TF_Shared --> SparkBatch
    TF_Shared --> Composer
    TF_Shared --> Metastore
    TF_Shared --> Notebook
    TF_MLOps_Shared --> Composer
    TF_MLOps --> VAI_Pipeline

    Composer --> MLOps_DAG
    Composer --> CellTower_DAG
    Composer --> Malware_DAG
    Composer --> Retail_DAG
    Scheduler --> Functions
    Functions --> SparkBatch
    VAI_Pipeline --> NotebookETL
    VAI_Pipeline --> NotebookML

    SparkBatch -->|reads/writes| GCS
    SparkBatch -->|writes| BQ
    NotebookETL --> BQ
    NotebookML --> VAI_Pipeline
    StreamingJob --> BQ
    DS_TS --> PythonETL
    DS_Churn --> ModelTrain
    DS_Parquet --> SparkBatch
    BQ -->|serves data| ModelTrain
    Metastore --> SparkBatch
    Metastore --> BQ

    %% Compute scripts into SparkBatch
    SparkBatch --> CommonUtils
    SparkBatch --> ModelTrain
    SparkBatch --> BatchScore
    SparkBatch --> ScalaJob
    SparkBatch --> PythonETL
    SparkBatch --> StreamingJob

    %% Metadata SQL
    Metastore --> Metastore_SQL

    %% Dependencies usage
    SparkBatch --> Docker_SM
    SparkBatch --> GraphFrames

    %% Templates usage
    VAI_Pipeline --> VAI_Template

    %% Click Events
    click TF_Shared "https://github.com/googlecloudplatform/serverless-spark-workshop/tree/main/s8s-prerequisites/00-scripts-and-config/terraform/"
    click TF_MLOps_Shared "https://github.com/googlecloudplatform/serverless-spark-workshop/tree/main/s8s-spark-mlops/00-env-setup-shared/terraform/"
    click TF_MLOps "https://github.com/googlecloudplatform/serverless-spark-workshop/tree/main/s8s-spark-mlops/00-env-setup/terraform/"
    click MLOps_DAG "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/02-scripts/airflow/pipeline.py"
    click CellTower_DAG "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/cell-tower-anomaly-detection/00-scripts/cell-tower-airflow.py"
    click Malware_DAG "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/malware_detection/00-scripts/malware_detection_airflow.py"
    click Retail_DAG "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/retail_store_analytics_metastore/00-scripts/retail-analytics-airflow.py"
    click Functions "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/02-scripts/cloud-functions/main.py"
    click CommonUtils "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/02-scripts/pyspark/common_utils.py"
    click ModelTrain "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/02-scripts/pyspark/model_training.py"
    click BatchScore "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/02-scripts/pyspark/batch_scoring.py"
    click ScalaJob "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/covid-economic-impact-scala/00-scripts/covid-economic-impact-assessment.scala"
    click PythonETL "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/covid-economic-impact-batches/00-scripts/covid_economic_impact_etl.py"
    click NotebookETL "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/covid-economic-impact-vertex-ai/00-scripts/covid_economic_impact_etl.ipynb"
    click NotebookML "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/customer-churn-prediction-vertex-ai/00-scripts/customer_churn_model_building.ipynb"
    click StreamingJob "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/serverless_spark_streaming/00-scripts/spark_streaming_invoice.py"
    click DS_TS "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/timeseries_forecasting/01-datasets/train.csv"
    click DS_Churn "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/01-datasets/customer_churn_train_data.csv"
    click DS_Parquet "https://github.com/googlecloudplatform/serverless-spark-workshop/tree/main/cell-tower-anomaly-detection-dbt/cell-tower-anomaly-detection-dbt/01-datasets/customers_raw_data/"
    click BQ_WC_Script "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/bigquery-shakespeare-word-count/00-scripts-and-config/wordcount-calculation-bigquery.py"
    click Metastore_SQL "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/retail_store_analytics_metastore_sparksql/00-scripts/retail_analytics_schema_definition_sparksql.sql"
    click Docker_SM "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/social_media_data_analytics/02-dependencies/Dockerfile.txt"
    click GraphFrames "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/social_network_graph/02-dependencies/graphframes-0.8.1-spark3.0-s_2.12.zip"
    click VAI_Template "https://github.com/googlecloudplatform/serverless-spark-workshop/blob/main/s8s-spark-mlops/04-templates/customer_churn_vai_pipeline_template.json"

    %% Styles
    classDef gcp fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef code fill:#E0FFD0,stroke:#333,stroke-width:1px
    classDef data fill:#FFE0B2,stroke:#333,stroke-width:1px
    classDef meta fill:#F4E1FF,stroke:#333,stroke-width:1px
    classDef deps fill:#F0F0F0,stroke:#333,stroke-width:1px
    classDef template fill:#F5F5F5,stroke:#333,stroke-width:1px
```
