How googlecloudplatform/serverless-spark-workshop works
Project Type and High‐Level Purpose
• Monorepo of Spark-based, Serverless pipelines and ML/analytics notebooks on GCP.
• Each subfolder under the root (e.g., cell-tower-anomaly-detection-dbt, timeseries_forecasting, s8s-spark-mlops, etc.) is a standalone solution accelerator demonstrating a common Spark use case (batch ETL, streaming, interactive ML, etc.).
File Structure Patterns
• Each accelerator contains:
– 00-scripts or 02-scripts: Spark job entrypoints (PySpark, Scala, SQL scripts, or notebooks).
– 01-datasets: sample data or pointers to GCS/BigQuery tables.
– 02-dependencies (if present): Dockerfiles or extra libs for containerized jobs.
– instructions folder: GCP setup steps (Terraform, Composer, BigQuery dataset creation).
• Root also holds Terraform configs for shared prerequisites (s8s-prerequisites/) and a global README listing all accelerators.
README Insights
• Common architecture:
– Infrastructure provisioning via Terraform → GCS buckets, BigQuery datasets, Dataproc Serverless endpoint, Dataproc Metastore, Vertex AI resources, Cloud Composer, Cloud Functions, Cloud Scheduler.
– Data ingestion: upload to GCS or directly to BigQuery/BigLake.
– Orchestration: Cloud Composer (Airflow DAGs), Cloud Scheduler + Cloud Functions, or manual gcloud commands.
– Compute: Dataproc Serverless Spark Batches or interactive Serverless Spark sessions within Vertex AI notebooks.
– Storage & catalog: BigQuery tables or BigLake external tables (Parquet/CSV) registered via Dataproc Metastore.
– Downstream uses: analytics dashboards, ML model training & serving in Vertex AI.
Main Components & Boundaries
a. User / Developer Environment
– Local IDE or Cloud Shell
– Vertex AI Workbench notebooks
– gcloud CLI
b. Infrastructure as Code (IaC)
– Terraform → provision GCS, BigQuery, Dataproc Serverless, Composer, Metastore, Vertex AI.
c. Orchestration Layer
– Cloud Composer (Airflow DAGs)
– Cloud Scheduler → triggers Cloud Functions → invokes Dataproc Serverless jobs
– Vertex AI Pipelines (for ML pipelines)
d. Compute Layer
– Dataproc Serverless Spark Batches (batch ETL, streaming, SQL, ML jobs)
– Interactive Serverless Spark on Vertex AI notebooks
e. Storage Layer
– Google Cloud Storage (raw data, Parquet/CSV staging)
– BigQuery / BigLake external tables (via Dataproc Metastore)
– Dataproc Metastore (Hive-compatible catalog)
f. ML & Serving
– Vertex AI Training & Endpoints (built‐in in some accelerators)
– Cloud Functions (e.g., for event‐driven triggers)
Relationships & Data Flows
– Data upload → GCS
– GCS → Dataproc Serverless (Spark reads staging data)
– Spark writes → BigQuery or back to GCS (transformed/parquet)
– BigQuery → BI or ML pipelines
– Orchestrator → triggers Compute
– IaC (Terraform) → provisions all resources
• For streaming solutions, show Pub/Sub or JSON files directory feeding Spark Structured Streaming.
Architectural Patterns
• ETL Batch Pattern: GCS → Spark Batch → BigQuery
• Streaming Pattern: Pub/Sub / GCS streaming source → Spark Structured Streaming → BigQuery
• Interactive ML Pattern: Vertex AI Notebook → Serverless Spark session → BigQuery → Vertex AI Training
• SQL–only Pattern: Spark SQL via Dataproc Serverless + Metastore → BigQuery
Technologies & Frameworks
• Dataproc Serverless Spark
• BigQuery & BigLake (external tables)
• Dataproc Metastore
• Cloud Composer (Airflow)
• Terraform for IaC
• Vertex AI Workbench & Pipelines
• Cloud Functions & Scheduler
• dbt (only in cell-tower-anomaly-detection-dbt)
Detail Is Key—Don’t Overthink
• Every distinct component you see in the file tree and README.
• Even if two accelerators use the same service, show them separately if they run in parallel or via different orchestrators.
• Always annotate any terraform modules or setup scripts as a “Provisioning” component.