How ero67/cloud-computing-project works
Project Type & Purpose
• This is a data pipeline project that orchestrates data processing flows using Prefect. It is deployed on a GKE (Google Kubernetes Engine) cluster, uses Terraform for provisioning GCP resources, and leverages GitHub Actions for CI/CD automation.
• The project integrates multiple components: a Prefect server and worker for workflow orchestration, Kubernetes for container orchestration, Terraform for infrastructure as code, and various GCP services (GCS, BigQuery, Cloud SQL).
File Structure Insights
• Top-Level CI/CD automation is set in the “.github/workflows” directory with different pipelines (Docker image builds, Terraform-based deployments, infrastructure updates).
• The “pipeline-project/k8s/base” directory contains Kubernetes manifests that deploy essential components (e.g., Prefect server, worker, cloudsql proxy, taxi processing job).
• The “pipeline-project/src/processing” folder holds the core data processing code with Prefect flow definitions.
• The “pipeline-project/terraform” directory contains IaC modules for BigQuery, Cloud SQL, and GCS.
Key Architectural Components
a. CI/CD Pipeline & Repository:
• GitHub Repository, including GitHub Actions workflows.
• The build and push process to Google Container Registry.
• Terraform workflows triggered via GitHub Actions.
b. Cloud Infrastructure (GCP):
• Provisioned resources: GCS buckets for raw/processed data, BigQuery, and Cloud SQL.
• Looker integration (for data visualization).
c. Kubernetes Cluster (GKE):
• Cluster details including nodes, k8s versions, and autoscaling details.
• The “data-pipeline” namespace containing:
– Prefect Server: Orchestrates and monitors workflow runs.
– Prefect Worker: Executes tasks and initiates Kubernetes jobs.
– Supporting deployments: Cloud SQL proxy, taxi-data-processing job.
– Services, ReplicaSets, and Pods for each deployment.
d. Data Pipeline Flow:
• The flow starts from the NYC Taxi Data Source, downloading/parquet conversion and then processing with the Prefect Flow.
• Data is stored in GCS (raw and processed) and loaded into BigQuery.
• Optionally, there’s future integration with Cloud SQL.
Architectural Patterns & Design Principles
• Separation of Concerns: Different layers for CI/CD, infrastructure provisioning, orchestration, and data processing.
• Infrastructure as Code (Terraform modules) ensures reproducibility.
• Container orchestration on Kubernetes provides scalability and fault isolation.
• The Prefect architecture demonstrates decoupling between the orchestration engine (Prefect server) and its workers via explicit work pool configurations.