# thebluesurf3r/gcp-data-pipeline architecture

> Architecture diagram of the GitHub repository thebluesurf3r/gcp-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thebluesurf3r/gcp-data-pipeline
- Repository: https://github.com/thebluesurf3r/gcp-data-pipeline

## Overview

This repository is a small GCP data pipeline: running `main.py` uploads a local CSV to a GCS landing path, then invokes PySpark to read, transform, and append records to BigQuery. The graph separates orchestration, ingestion, transformation, and configuration. Source confirms the landing read and BigQuery write; the README describes the end-to-end workflow. External cloud services are shown as endpoints, not as inspected implementation details.

Main entry points: [`src/ingest.py`](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/ingest.py) (GCS uploader), [`src/pyspark_transform.py`](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/pyspark_transform.py) (Spark transformation), [`main.py`](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/main.py) (Pipeline runner), [`src/utils.py`](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/utils.py) (Storage client helper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Pipeline runner<br/>[main.py]"]
end

subgraph group_ingestion["Ingestion"]
  node_ingest["GCS uploader<br/>[ingest.py]"]
  node_storage_utils["Storage client helper<br/>[utils.py]"]
end

subgraph group_transformation["Transformation"]
  node_transform["Spark transformation"]
end

subgraph group_configuration["Configuration"]
  node_config["GCP settings<br/>[gcp_config.py]"]
end

node_operator(("Pipeline operator"))
node_local_csv["Local CSV"]
node_gcs_landing[("GCS landing bucket")]
node_gcs_temp[("GCS temporary bucket")]
node_bigquery[("BigQuery table")]

node_operator -->|"runs"| node_main
node_main -->|"calls"| node_ingest
node_main -->|"selects path"| node_local_csv
node_main -->|"calls"| node_transform
node_ingest -->|"gets client"| node_storage_utils
node_storage_utils -->|"reads project"| node_config
node_storage_utils -->|"creates client"| node_gcs_landing
node_ingest -->|"uploads file"| node_local_csv
node_ingest -->|"writes landing file"| node_gcs_landing
node_ingest -->|"reads bucket"| node_config
node_transform -->|"reads CSV"| node_gcs_landing
node_transform -->|"appends rows"| node_bigquery
node_transform -->|"uses temporary storage"| node_gcs_temp
node_transform -->|"reads Spark and table settings"| node_config

click node_main "https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/main.py"
click node_ingest "https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/ingest.py"
click node_storage_utils "https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/utils.py"
click node_transform "https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/pyspark_transform.py"
click node_config "https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/config/gcp_config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main toneBlue
class node_ingest,node_storage_utils,node_gcs_landing,node_gcs_temp,node_bigquery toneAmber
class node_transform toneMint
class node_config toneRose
class node_operator,node_local_csv toneIndigo
```

## Components

### Orchestration

- [Pipeline runner](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/main.py): `main.py`

### Ingestion

- [GCS uploader](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/ingest.py): `src/ingest.py`
- [Storage client helper](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/utils.py): `src/utils.py`

### Transformation

- [Spark transformation](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/src/pyspark_transform.py): `src/pyspark_transform.py`

### Configuration

- [GCP settings](https://github.com/thebluesurf3r/gcp-data-pipeline/blob/main/config/gcp_config.py): `config/gcp_config.py`

### Other components

- **Pipeline operator**
- **Local CSV**
- **GCS landing bucket**
- **GCS temporary bucket**
- **BigQuery table**

## Connections

- Pipeline operator → Pipeline runner: runs
- Pipeline runner → GCS uploader: calls
- Pipeline runner → Local CSV: selects path
- Pipeline runner → Spark transformation: calls
- GCS uploader → Storage client helper: gets client
- Storage client helper → GCP settings: reads project
- Storage client helper → GCS landing bucket: creates client
- GCS uploader → Local CSV: uploads file
- GCS uploader → GCS landing bucket: writes landing file
- GCS uploader → GCP settings: reads bucket
- Spark transformation → GCS landing bucket: reads CSV
- Spark transformation → BigQuery table: appends rows
- Spark transformation → GCS temporary bucket: uses temporary storage
- Spark transformation → GCP settings: reads Spark and table settings