manojvsj/gcp-data-engineering-projects

1. Repository shape

- This is a monorepo of eight standalone GCP mini-projects, not a single application.

- Each numbered directory is an independent architecture demo with its own Python code, setup/deploy/cleanup scripts, requirements, and in some cases sample data or container definitions.

- The repo’s main architectural boundary is therefore project-level isolation: each folder can be diagrammed separately, while the top-level README describes the progression across patterns.

2. Shared architectural theme

- Most projects follow a common cloud data engineering pattern: source data or events enter GCP, a compute layer transforms/validates them, and a serving/storage layer persists or exposes results.

- The main GCP services used across the repo are Cloud Storage, Pub/Sub, Dataflow, Cloud Functions, Dataproc, BigQuery, Cloud Run, and Vertex AI.

- Several projects also introduce Apache Spark and Apache Iceberg for lakehouse-style storage and table management.

3. Project 1: event-driven file processing

- This is the simplest event-driven flow: GCS upload → Cloud Function → BigQuery load.

- The trigger boundary is object creation in Cloud Storage; the function acts as the processing gate and writes into BigQuery.

- This design is optimized for file arrival events and batch-like ingestion of CSVs, with schema inference handled by BigQuery.

4. Project 2: real-time data ingestion

- The streaming path is publisher → Pub/Sub → Dataflow (Apache Beam) → BigQuery.

- publisher.py is the event producer, while streaming_pipeline.py represents the streaming transform/load layer.

- The key boundary here is the separation between message ingestion and stream processing, with BigQuery as the analytical sink.

- The presence of local setup scripts suggests the pipeline is intended to be tested both locally and in cloud deployment.

5. Project 3: scheduled weather batch processing

- This project is a time-based ingestion pattern: Cloud Scheduler triggers Cloud Function → Open-Meteo API → BigQuery.

- The external dependency is an API call rather than an uploaded file or message stream.

- The data flow is append-oriented, building a historical weather dataset in BigQuery with UTC timestamps for temporal consistency.

6. Project 4: Dataproc Spark processing

- This is a classic batch ETL pipeline: input CSV in GCS/local data → PySpark on Dataproc → partitioned Parquet in GCS → BigQuery external table.

- The repo includes a Spark job, a Dataproc runner, a BigQuery SQL script, and a GCS connector JAR, which together define the batch compute and storage integration.

- The architecture boundary is between distributed processing on Dataproc and query serving via BigQuery external tables.

7. Project 5: data quality checks framework

- This project is a scheduled validation layer over warehouse data: Cloud Scheduler / Cloud Function → BigQuery table scan → quality metrics/reporting.

- dq-config.yaml indicates checks are configuration-driven rather than hardcoded, which is important architecturally: the framework is designed as a reusable validation engine.

- The checks focus on common warehouse issues such as nulls, duplicates, and range validation, with partition awareness for targeted assessment.

8. Project 6: Dataproc Spark + Iceberg medallion architecture

- This introduces a multi-hop lakehouse flow: raw input data → Spark jobs on Dataproc → Bronze/Silver/Gold Iceberg tables → BigQuery external access.

- The medallion boundary is explicit: raw ingestion, cleaned/curated transformations, then analytics-ready outputs.

- Apache Iceberg provides the table abstraction, enabling ACID behavior, schema evolution, and time travel; Dataproc Metastore manages table metadata.

- The repo also includes a BigQuery SQL script, showing that the serving layer can be queried outside Spark.

9. Project 7: Iceberg + BigQuery lambda architecture

- This is the most complex data-flow design in the repo, combining speed and batch layers.

- Speed path: Pub/Sub → Dataflow → BigQuery for near-real-time analytics.

- Batch path: Pub/Sub → Dataflow → GCS → Spark on Dataproc Serverless → Iceberg for deeper historical processing.

- The serving boundary is split: BigQuery serves fast operational queries, while Iceberg serves richer analytical workloads.

- The repository contains separate subfolders for Pub/Sub, Dataflow, Spark batch jobs, and configuration, reflecting a deliberately modular architecture.

10. Project 8: natural language to BigQuery with Vertex AI and MCP

- This is an AI-assisted query architecture: Streamlit UI → MCP BigQuery server → BigQuery, with Vertex AI Gemini generating or assisting with SQL.

- The architecture includes a secure execution boundary via the Model Context Protocol pattern, so the assistant and query execution are separated.

- The repo uses containerization and deployment scripts for local/cloud execution, suggesting a service-oriented setup rather than a single monolith.

- The main interaction pattern is not ETL but natural language request → validated query → warehouse execution → summarized response.

11. Cross-cutting infrastructure and deployment model

- Most projects include setup, deploy, and cleanup scripts, which implies each demo is intended to be reproducible and disposable in GCP.

- Credentials are managed through a standard service account JSON key via GOOGLE_APPLICATION_CREDENTIALS.

- Some projects use Docker, Cloud Run, or containerized components, but the dominant repo style is still Python-based cloud automation with GCP-native services.

12. How to diagram the repo accurately

- At the top level, diagram this as a portfolio of independent reference architectures, not one end-to-end pipeline.

- For each project, show:

- the trigger/source,

- the processing engine,

- the storage/serving layer,

- and any external service such as Open-Meteo or Vertex AI.

- The important architectural distinction across the repo is the evolution from event-driven and scheduled ingestion, to streaming, to batch lakehouse, to hybrid lambda, and finally to an AI-enabled query interface.