# manojvsj/gcp-data-engineering-projects architecture

> Architecture diagram of the GitHub repository manojvsj/gcp-data-engineering-projects, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/manojvsj/gcp-data-engineering-projects
- Repository: https://github.com/manojvsj/gcp-data-engineering-projects

## Overview

1. **Repository shape**
   - This is a **monorepo of eight standalone GCP mini-projects**, not a single application.
   - Each numbered directory is an independent architecture demo with its own Python code, setup/deploy/cleanup scripts, requirements, and in some cases sample data or container definitions.
   - The repo’s main architectural boundary is therefore **project-level isolation**: each folder can be diagrammed separately, while the top-level README describes the progression across patterns.

2. **Shared architectural theme**
   - Most projects follow a common cloud data engineering pattern: **source data or events enter GCP, a compute layer transforms/validates them, and a serving/storage layer persists or exposes results**.
   - The main GCP services used across the repo are **Cloud Storage, Pub/Sub, Dataflow, Cloud Functions, Dataproc, BigQuery, Cloud Run, and Vertex AI**.
   - Several projects also introduce **Apache Spark** and **Apache Iceberg** for lakehouse-style storage and table management.

3. **Project 1: event-driven file processing**
   - This is the simplest event-driven flow: **GCS upload → Cloud Function → BigQuery load**.
   - The trigger boundary is object creation in Cloud Storage; the function acts as the processing gate and writes into BigQuery.
   - This design is optimized for **file arrival events** and batch-like ingestion of CSVs, with schema inference handled by BigQuery.

4. **Project 2: real-time data ingestion**
   - The streaming path is **publisher → Pub/Sub → Dataflow (Apache Beam) → BigQuery**.
   - `publisher.py` is the event producer, while `streaming_pipeline.py` represents the streaming transform/load layer.
   - The key boundary here is the separation between **message ingestion** and **stream processing**, with BigQuery as the analytical sink.
   - The presence of local setup scripts suggests the pipeline is intended to be tested both locally and in cloud deployment.

5. **Project 3: scheduled weather batch processing**
   - This project is a time-based ingestion pattern: **Cloud Scheduler triggers Cloud Function → Open-Meteo API → BigQuery**.
   - The external dependency is an API call rather than an uploaded file or message stream.
   - The data flow is append-oriented, building a historical weather dataset in BigQuery with UTC timestamps for temporal consistency.

6. **Project 4: Dataproc Spark processing**
   - This is a classic batch ETL pipeline: **input CSV in GCS/local data → PySpark on Dataproc → partitioned Parquet in GCS → BigQuery external table**.
   - The repo includes a Spark job, a Dataproc runner, a BigQuery SQL script, and a GCS connector JAR, which together define the batch compute and storage integration.
   - The architecture boundary is between **distributed processing on Dataproc** and **query serving via BigQuery external tables**.

7. **Project 5: data quality checks framework**
   - This project is a scheduled validation layer over warehouse data: **Cloud Scheduler / Cloud Function → BigQuery table scan → quality metrics/reporting**.
   - `dq-config.yaml` indicates checks are configuration-driven rather than hardcoded, which is important architecturally: the framework is designed as a **reusable validation engine**.
   - The checks focus on common warehouse issues such as nulls, duplicates, and range validation, with partition awareness for targeted assessment.

8. **Project 6: Dataproc Spark + Iceberg medallion architecture**
   - This introduces a multi-hop lakehouse flow: **raw input data → Spark jobs on Dataproc → Bronze/Silver/Gold Iceberg tables → BigQuery external access**.
   - The medallion boundary is explicit: raw ingestion, cleaned/curated transformations, then analytics-ready outputs.
   - Apache Iceberg provides the table abstraction, enabling ACID behavior, schema evolution, and time travel; Dataproc Metastore manages table metadata.
   - The repo also includes a BigQuery SQL script, showing that the serving layer can be queried outside Spark.

9. **Project 7: Iceberg + BigQuery lambda architecture**
   - This is the most complex data-flow design in the repo, combining **speed and batch layers**.
   - **Speed path:** Pub/Sub → Dataflow → BigQuery for near-real-time analytics.
   - **Batch path:** Pub/Sub → Dataflow → GCS → Spark on Dataproc Serverless → Iceberg for deeper historical processing.
   - The serving boundary is split: **BigQuery serves fast operational queries**, while **Iceberg serves richer analytical workloads**.
   - The repository contains separate subfolders for Pub/Sub, Dataflow, Spark batch jobs, and configuration, reflecting a deliberately modular architecture.

10. **Project 8: natural language to BigQuery with Vertex AI and MCP**
   - This is an AI-assisted query architecture: **Streamlit UI → MCP BigQuery server → BigQuery**, with **Vertex AI Gemini** generating or assisting with SQL.
   - The architecture includes a secure execution boundary via the **Model Context Protocol pattern**, so the assistant and query execution are separated.
   - The repo uses containerization and deployment scripts for local/cloud execution, suggesting a service-oriented setup rather than a single monolith.
   - The main interaction pattern is not ETL but **natural language request → validated query → warehouse execution → summarized response**.

11. **Cross-cutting infrastructure and deployment model**
   - Most projects include **setup, deploy, and cleanup scripts**, which implies each demo is intended to be reproducible and disposable in GCP.
   - Credentials are managed through a standard **service account JSON key via `GOOGLE_APPLICATION_CREDENTIALS`**.
   - Some projects use Docker, Cloud Run, or containerized components, but the dominant repo style is still **Python-based cloud automation** with GCP-native services.

12. **How to diagram the repo accurately**
   - At the top level, diagram this as a **portfolio of independent reference architectures**, not one end-to-end pipeline.
   - For each project, show:
     - the **trigger/source**,
     - the **processing engine**,
     - the **storage/serving layer**,
     - and any **external service** such as Open-Meteo or Vertex AI.
   - The important architectural distinction across the repo is the evolution from **event-driven and scheduled ingestion**, to **streaming**, to **batch lakehouse**, to **hybrid lambda**, and finally to an **AI-enabled query interface**.

Main entry points: [`README.md`](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/README.md) (Repo portfolio), [`3.weather-batch-processing/main.py`](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/3.weather-batch-processing/main.py) (Weather sync), [`7.iceberg-bigquery-lambda-architecture/dataflow_jobs/streaming_pipeline.py`](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/dataflow_jobs/streaming_pipeline.py) (Lambda router), [`8.bigquery-vertexai-mcp-connection/mcp-bigquery-server/main.py`](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/8.bigquery-vertexai-mcp-connection/mcp-bigquery-server/main.py) (MCP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

node_repo["Repo portfolio<br/>monorepo<br/>[README.md]"]
node_p1_source(("File upload<br/>GCS trigger<br/>[main.py]"))
node_p1_fn["Load function<br/>Cloud Function<br/>[main.py]"]
node_p1_bq[("Warehouse sink<br/>BigQuery load")]
node_p2_pub(("Publisher<br/>event producer<br/>[publisher.py]"))
node_p2_df["Stream processor<br/>Dataflow Beam"]
node_p2_bq[("Streaming sink<br/>BigQuery")]
node_p3_sched["Weather sync<br/>Scheduler + Function<br/>[main.py]"]
node_p3_api(("Open-Meteo<br/>external API"))
node_p3_bq[("Weather table<br/>BigQuery append")]
node_p4_spark["Spark ETL<br/>Dataproc PySpark<br/>[pyspark_job.py]"]
node_p4_gcs[("Parquet lake<br/>GCS storage")]
node_p4_bq[("External view<br/>BigQuery external table")]
node_p5_dq["DQ engine<br/>config-driven checks<br/>[dq_check.py]"]
node_p5_cfg["DQ config<br/>YAML rules<br/>[dq-config.yaml]"]
node_p5_bq[("Quality scan<br/>BigQuery source")]
node_p6_raw["Raw input<br/>landing data<br/>[orders_data.csv]"]
node_p6_spark["Medallion ETL<br/>Dataproc Spark<br/>[pyspark_job.py]"]
node_p6_iceberg[("Iceberg tables<br/>lakehouse storage<br/>[config.yaml]")]
node_p6_bq[("BQ access<br/>BigQuery SQL")]
node_p7_pub(("Pub/Sub ingress<br/>event stream<br/>[publisher.py]"))
node_p7_df["Lambda router<br/>Dataflow streaming"]
node_p7_bq[("Speed layer<br/>BigQuery")]
node_p7_batch["Batch lakehouse<br/>Spark on Dataproc Serverless"]
node_p7_iceberg[("Historical store<br/>Iceberg tables")]
node_p8_ui["Chat UI<br/>Streamlit app<br/>[app.py]"]
node_p8_mcp["MCP server<br/>BigQuery tool server<br/>[main.py]"]
node_p8_ai(("Gemini assist<br/>Vertex AI"))
node_p8_bq[("Query warehouse<br/>BigQuery")]
node_p9_agent["GCP agent<br/>AI assistant<br/>[agent.py]"]
node_p9_tools["Agent tools<br/>BQ/GCS tools<br/>[bq_tools.py]"]

node_repo --> node_p1_source
node_repo --> node_p2_pub
node_repo --> node_p3_sched
node_repo --> node_p4_spark
node_repo --> node_p5_dq
node_repo --> node_p6_spark
node_repo --> node_p7_pub
node_repo --> node_p8_ui
node_repo --> node_p9_agent
node_p1_source --> node_p1_fn
node_p1_fn --> node_p1_bq
node_p2_pub --> node_p2_df
node_p2_df --> node_p2_bq
node_p3_sched --> node_p3_api
node_p3_sched --> node_p3_bq
node_p4_spark --> node_p4_gcs
node_p4_gcs --> node_p4_bq
node_p5_cfg --> node_p5_dq
node_p5_bq --> node_p5_dq
node_p6_raw --> node_p6_spark
node_p6_spark --> node_p6_iceberg
node_p6_iceberg --> node_p6_bq
node_p7_pub --> node_p7_df
node_p7_df --> node_p7_bq
node_p7_df --> node_p7_batch
node_p7_batch --> node_p7_iceberg
node_p8_ui --> node_p8_mcp
node_p8_mcp --> node_p8_ai
node_p8_mcp --> node_p8_bq
node_p9_agent --> node_p9_tools

click node_repo "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/README.md"
click node_p1_source "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/1.event-driven-file-processing/main.py"
click node_p1_fn "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/1.event-driven-file-processing/main.py"
click node_p2_pub "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/2.realtime-data-ingestion/publisher.py"
click node_p2_df "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/2.realtime-data-ingestion/streaming_pipeline.py"
click node_p3_sched "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/3.weather-batch-processing/main.py"
click node_p4_spark "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/4.dataproc-spark-processing/pyspark_job.py"
click node_p4_bq "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/4.dataproc-spark-processing/bigquery_script.sql"
click node_p5_dq "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/5.data-quality-checks-framework/dq_check.py"
click node_p5_cfg "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/5.data-quality-checks-framework/dq-config.yaml"
click node_p6_raw "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/data/input/orders_data.csv"
click node_p6_spark "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/pyspark_jobs/pyspark_job.py"
click node_p6_iceberg "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/pyspark_jobs/config.yaml"
click node_p6_bq "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/bigquery_script.sql"
click node_p7_pub "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/pub_sub/publisher.py"
click node_p7_df "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/dataflow_jobs/streaming_pipeline.py"
click node_p7_batch "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/spark_jobs/gcs_to_iceberg_batch.py"
click node_p8_ui "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/8.bigquery-vertexai-mcp-connection/streamlit-app/app.py"
click node_p8_mcp "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/8.bigquery-vertexai-mcp-connection/mcp-bigquery-server/main.py"
click node_p9_agent "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/9.google-agent-sdk-gcp-assistant/agent/agent.py"
click node_p9_tools "https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/9.google-agent-sdk-gcp-assistant/agent/tools/bq_tools.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_repo,node_p1_source,node_p1_fn,node_p1_bq,node_p2_pub,node_p2_df,node_p2_bq,node_p3_sched,node_p3_api,node_p3_bq,node_p4_spark,node_p4_gcs,node_p4_bq,node_p5_dq,node_p5_cfg,node_p5_bq,node_p6_raw,node_p6_spark,node_p6_iceberg,node_p6_bq,node_p7_pub,node_p7_df,node_p7_bq,node_p7_batch,node_p7_iceberg,node_p8_ui,node_p8_mcp,node_p8_ai,node_p8_bq,node_p9_agent,node_p9_tools toneNeutral
```

## Components

### Other components

- [Repo portfolio](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/README.md): monorepo · Collection of independent GCP reference architectures · `README.md`
- [File upload](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/1.event-driven-file-processing/main.py): GCS trigger · CSV arrival boundary for event-driven ingestion · `1.event-driven-file-processing/main.py`
- [Load function](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/1.event-driven-file-processing/main.py): Cloud Function · Transforms uploaded files and loads rows to BigQuery · `1.event-driven-file-processing/main.py`
- **Warehouse sink**: BigQuery load · Analytical destination for uploaded file data
- [Publisher](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/2.realtime-data-ingestion/publisher.py): event producer · Emits streaming events into Pub/Sub · `2.realtime-data-ingestion/publisher.py`
- [Stream processor](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/2.realtime-data-ingestion/streaming_pipeline.py): Dataflow Beam · Consumes messages and transforms them in flight · `2.realtime-data-ingestion/streaming_pipeline.py`
- **Streaming sink**: BigQuery · Real-time analytical store
- [Weather sync](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/3.weather-batch-processing/main.py): Scheduler + Function · Time-based trigger that fetches external weather data · `3.weather-batch-processing/main.py`
- **Open-Meteo**: external API · Source of weather observations
- **Weather table**: BigQuery append · Historical weather dataset with UTC timestamps
- [Spark ETL](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/4.dataproc-spark-processing/pyspark_job.py): Dataproc PySpark · Batch transforms CSV input into partitioned parquet · `4.dataproc-spark-processing/pyspark_job.py`
- **Parquet lake**: GCS storage · Partitioned output used for downstream querying
- [External view](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/4.dataproc-spark-processing/bigquery_script.sql): BigQuery external table · SQL access over parquet in object storage · `4.dataproc-spark-processing/bigquery_script.sql`
- [DQ engine](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/5.data-quality-checks-framework/dq_check.py): config-driven checks · Scans warehouse data using reusable validation rules · `5.data-quality-checks-framework/dq_check.py`
- [DQ config](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/5.data-quality-checks-framework/dq-config.yaml): YAML rules · Defines null, duplicate, and range checks · `5.data-quality-checks-framework/dq-config.yaml`
- **Quality scan**: BigQuery source · Warehouse tables under validation
- [Raw input](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/data/input/orders_data.csv): landing data · Source CSV data entering the lakehouse pipeline · `6.dataproc-spark-iceberg-medallion/data/input/orders_data.csv`
- [Medallion ETL](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/pyspark_jobs/pyspark_job.py): Dataproc Spark · Builds bronze, silver, and gold Iceberg tables · `6.dataproc-spark-iceberg-medallion/pyspark_jobs/pyspark_job.py`
- [Iceberg tables](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/pyspark_jobs/config.yaml): lakehouse storage · ACID table layer for curated analytics · `6.dataproc-spark-iceberg-medallion/pyspark_jobs/config.yaml`
- [BQ access](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/6.dataproc-spark-iceberg-medallion/bigquery_script.sql): BigQuery SQL · Query serving over the lakehouse outputs · `6.dataproc-spark-iceberg-medallion/bigquery_script.sql`
- [Pub/Sub ingress](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/pub_sub/publisher.py): event stream · Shared event source for speed and batch paths · `7.iceberg-bigquery-lambda-architecture/pub_sub/publisher.py`
- [Lambda router](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/dataflow_jobs/streaming_pipeline.py): Dataflow streaming · Splits stream into low-latency and batch paths · `7.iceberg-bigquery-lambda-architecture/dataflow_jobs/streaming_pipeline.py`
- **Speed layer**: BigQuery · Near-real-time serving store
- [Batch lakehouse](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/7.iceberg-bigquery-lambda-architecture/spark_jobs/gcs_to_iceberg_batch.py): Spark on Dataproc Serverless · Loads curated history into Iceberg · `7.iceberg-bigquery-lambda-architecture/spark_jobs/gcs_to_iceberg_batch.py`
- **Historical store**: Iceberg tables · Batch analytical layer for deeper processing
- [Chat UI](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/8.bigquery-vertexai-mcp-connection/streamlit-app/app.py): Streamlit app · Natural language interface for warehouse questions · `8.bigquery-vertexai-mcp-connection/streamlit-app/app.py`
- [MCP server](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/8.bigquery-vertexai-mcp-connection/mcp-bigquery-server/main.py): BigQuery tool server · Validated execution boundary for SQL access · `8.bigquery-vertexai-mcp-connection/mcp-bigquery-server/main.py`
- **Gemini assist**: Vertex AI · Generates or helps refine SQL requests
- **Query warehouse**: BigQuery · Execution target for assistant-driven queries
- [GCP agent](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/9.google-agent-sdk-gcp-assistant/agent/agent.py): AI assistant · Agent SDK-based assistant for cloud tools and data access · `9.google-agent-sdk-gcp-assistant/agent/agent.py`
- [Agent tools](https://github.com/manojvsj/gcp-data-engineering-projects/blob/main/9.google-agent-sdk-gcp-assistant/agent/tools/bq_tools.py): BQ/GCS tools · Tooling layer for warehouse and object storage actions · `9.google-agent-sdk-gcp-assistant/agent/tools/bq_tools.py`

## Connections

- Repo portfolio → File upload: Project 1 architecture
- Repo portfolio → Publisher: Project 2 architecture
- Repo portfolio → Weather sync: Project 3 architecture
- Repo portfolio → Spark ETL: Project 4 architecture
- Repo portfolio → DQ engine: Project 5 architecture
- Repo portfolio → Medallion ETL: Project 6 architecture
- Repo portfolio → Pub/Sub ingress: Project 7 architecture
- Repo portfolio → Chat UI: Project 8 architecture
- Repo portfolio → GCP agent: Project 9 architecture
- File upload → Load function: Object creation triggers function
- Load function → Warehouse sink: Loads parsed rows
- Publisher → Stream processor: Publishes streaming events
- Stream processor → Streaming sink: Writes processed records
- Weather sync → Open-Meteo: Calls weather API
- Weather sync → Weather table: Appends weather facts
- Spark ETL → Parquet lake: Writes parquet outputs
- Parquet lake → External view: Exposes data for SQL access
- DQ config → DQ engine: Rules drive validation engine
- Quality scan → DQ engine: Scans warehouse tables
- Raw input → Medallion ETL: Raw data feeds medallion jobs
- Medallion ETL → Iceberg tables: Builds bronze silver gold tables
- Iceberg tables → BQ access: Supports SQL serving
- Pub/Sub ingress → Lambda router: Stream ingestion
- Lambda router → Speed layer: Speed path output
- Lambda router → Batch lakehouse: Batch handoff via GCS
- Batch lakehouse → Historical store: Curates historical lakehouse
- Chat UI → MCP server: Natural language requests
- MCP server → Gemini assist: Uses Gemini for SQL assistance
- MCP server → Query warehouse: Executes validated queries
- GCP agent → Agent tools: Agent delegates cloud actions