# ero67/cloud-computing-project architecture

> Architecture diagram of the GitHub repository ero67/cloud-computing-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ero67/cloud-computing-project
- Repository: https://github.com/ero67/cloud-computing-project

## Overview

##### Project Type & Purpose
- This is a data pipeline project that orchestrates data processing flows using Prefect. It is deployed on a GKE (Google Kubernetes Engine) cluster, uses Terraform for provisioning GCP resources, and leverages GitHub Actions for CI/CD automation.
- The project integrates multiple components: a Prefect server and worker for workflow orchestration, Kubernetes for container orchestration, Terraform for infrastructure as code, and various GCP services (GCS, BigQuery, Cloud SQL).

##### File Structure Insights
- Top-Level CI/CD automation is set in the “.github/workflows” directory with different pipelines (Docker image builds, Terraform-based deployments, infrastructure updates).
- The “pipeline-project/k8s/base” directory contains Kubernetes manifests that deploy essential components (e.g., Prefect server, worker, cloudsql proxy, taxi processing job).
- The “pipeline-project/src/processing” folder holds the core data processing code with Prefect flow definitions.
- The “pipeline-project/terraform” directory contains IaC modules for BigQuery, Cloud SQL, and GCS.

##### Key Architectural Components
a. CI/CD Pipeline & Repository:
- GitHub Repository, including GitHub Actions workflows.
- The build and push process to Google Container Registry.
- Terraform workflows triggered via GitHub Actions.
b. Cloud Infrastructure (GCP):
- Provisioned resources: GCS buckets for raw/processed data, BigQuery, and Cloud SQL.
- Looker integration (for data visualization).
c. Kubernetes Cluster (GKE):
- Cluster details including nodes, k8s versions, and autoscaling details.
- The “data-pipeline” namespace containing:
– Prefect Server: Orchestrates and monitors workflow runs.
– Prefect Worker: Executes tasks and initiates Kubernetes jobs.
– Supporting deployments: Cloud SQL proxy, taxi-data-processing job.
– Services, ReplicaSets, and Pods for each deployment.
d. Data Pipeline Flow:
- The flow starts from the NYC Taxi Data Source, downloading/parquet conversion and then processing with the Prefect Flow.
- Data is stored in GCS (raw and processed) and loaded into BigQuery.
- Optionally, there’s future integration with Cloud SQL.

##### Architectural Patterns & Design Principles
- Separation of Concerns: Different layers for CI/CD, infrastructure provisioning, orchestration, and data processing.
- Infrastructure as Code (Terraform modules) ensures reproducibility.
- Container orchestration on Kubernetes provides scalability and fault isolation.
- The Prefect architecture demonstrates decoupling between the orchestration engine (Prefect server) and its workers via explicit work pool configurations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CI/CD Pipeline & Repository
    subgraph "CI/CD Pipeline & Repository"
        GitHubRepo["GitHub Repository"]:::cicd
        GitHubActions["GitHub Actions Workflows"]:::cicd
        ContainerRegistry["Container Registry"]:::cicd
        GitHubRepo -->|"triggers"| GitHubActions
        GitHubActions -->|"Build&PUSH"| ContainerRegistry
    end

    %% Infrastructure Provisioning (Terraform)
    subgraph "Infrastructure Provisioning"
        TerraformModules["Terraform IaC Modules"]:::iac
        GCSRaw["GCS: Raw Data Bucket"]:::gcp
        GCSProcessed["GCS: Processed Data Bucket"]:::gcp
        BigQuery["BigQuery"]:::gcp
        CloudSQL["Cloud SQL"]:::gcp
        Looker["Looker Integration"]:::gcp
        TerraformModules -->|"Provisions"| GCSRaw
        TerraformModules -->|"Provisions"| GCSProcessed
        TerraformModules -->|"Provisions"| BigQuery
        TerraformModules -->|"Provisions"| CloudSQL
        TerraformModules -->|"Provisions"| Looker
    end

    %% Kubernetes Cluster (GKE)
    subgraph "Kubernetes Cluster (GKE)"
        K8sNamespace["data-pipeline Namespace"]
        subgraph " " 
            PrefectServer["Prefect Server"]:::kube
            PrefectWorker["Prefect Worker"]:::kube
            CloudSQLProxy["Cloud SQL Proxy"]:::kube
            TaxiJob["Taxi Data Processing Job"]:::kube
        end
        K8sNamespace --- PrefectServer
        K8sNamespace --- PrefectWorker
        K8sNamespace --- CloudSQLProxy
        K8sNamespace --- TaxiJob
        PrefectServer -->|"orchestrates"| PrefectWorker
        PrefectWorker -->|"triggers"| TaxiJob
    end

    %% Data Pipeline Flow
    subgraph "Data Pipeline Flow"
        NYCSource["NYC Taxi Data Source"]:::data
        PrefectFlow["Prefect Flow (Data Processing)"]:::data
        NYCSource -->|"download/convert"| PrefectFlow
        PrefectFlow -->|"store_raw"| GCSRaw
        PrefectFlow -->|"store_processed"| GCSProcessed
        GCSProcessed -->|"load"| BigQuery
        PrefectFlow -->|"optional"| CloudSQL
    end

    %% Deployment Flow from CI/CD to Kubernetes
    ContainerRegistry -->|"Deploys"| K8sNamespace

    %% Click Events
    click GitHubActions "https://github.com/ero67/cloud-computing-project/tree/main/.github/workflows"
    click K8sNamespace "https://github.com/ero67/cloud-computing-project/tree/main/pipeline-project/k8s/base"
    click PrefectFlow "https://github.com/ero67/cloud-computing-project/tree/main/pipeline-project/src/processing/flows"
    click TerraformModules "https://github.com/ero67/cloud-computing-project/tree/main/pipeline-project/terraform"

    %% Style Definitions
    classDef cicd fill:#6a0dad,stroke:#333,stroke-width:2px,color:#ffffff;
    classDef gcp fill:#1a73e8,stroke:#ffffff,stroke-width:2px,color:#ffffff;
    classDef kube fill:#ffa500,stroke:#333,stroke-width:2px,color:#000000;
    classDef data fill:#008000,stroke:#333,stroke-width:2px,color:#ffffff;
    classDef iac fill:#ff7f50,stroke:#333,stroke-width:2px,color:#ffffff;
```
