How thebluesurf3r/gcp-data-pipeline works
This repository is a small GCP data pipeline: running main.py uploads a local CSV to a GCS landing path, then invokes PySpark to read, transform, and append records to BigQuery. The graph separates orchestration, ingestion, transformation, and configuration. Source confirms the landing read and BigQuery write; the README describes the end-to-end workflow. External cloud services are shown as endpoints, not as inspected implementation details.
Main entry points: ingest.py GCS uploader · pyspark_transform.py Spark transformation · main.py Pipeline runner · utils.py Storage client helper
Components 10
Orchestration
- Pipeline runner
main.py
Ingestion
- GCS uploader
src/ingest.py - Storage client helper
src/utils.py
Transformation
- Spark transformation
src/pyspark_transform.py
Configuration
- GCP settings
config/gcp_config.py
Other components
- Pipeline operator
- Local CSV
- GCS landing bucket
- GCS temporary bucket
- BigQuery table
Connections 14
- Pipeline operator → Pipeline runner: runs
- Pipeline runner → GCS uploader: calls
- Pipeline runner → Local CSV: selects path
- Pipeline runner → Spark transformation: calls
- GCS uploader → Storage client helper: gets client
- Storage client helper → GCP settings: reads project
- Storage client helper → GCS landing bucket: creates client
- GCS uploader → Local CSV: uploads file
- GCS uploader → GCS landing bucket: writes landing file
- GCS uploader → GCP settings: reads bucket
- Spark transformation → GCS landing bucket: reads CSV
- Spark transformation → BigQuery table: appends rows
- Spark transformation → GCS temporary bucket: uses temporary storage
- Spark transformation → GCP settings: reads Spark and table settings