# a-vijai/gcp_project architecture

> Architecture diagram of the GitHub repository a-vijai/gcp_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/a-vijai/gcp_project
- Repository: https://github.com/a-vijai/gcp_project

## Overview

##### Project type and purpose
- This is a GCP-based data-pipeline project. It consists of:
- An Airflow job (airflow_job/airflow_job.py) for orchestration.
- A Spark job (spark_job/spark_transformation_job.py) for data transformation.
- Environment-specific configuration files (variables/dev/variables.json and variables/prod/variables.json).
- A CI/CD pipeline defined in .github/workflows/ci-cd.yaml to deploy Airflow DAGs and Spark jobs to GCP.

##### File structure
- Top-level:
- .github/workflows/ci-cd.yaml – CI/CD workflow.
- airflow_job/ – contains the Airflow DAG or operator code.
- spark_job/ – contains the Spark transformation logic.
- variables/dev and variables/prod – JSON config files holding parameters for each environment.
- There are no separate “frontend” or “backend” folders—this is purely a data orchestration and processing project.
- CI/CD files indicate automated deployment of jobs to GCP.

##### README for additional insights
- The README is minimal; it only names the project as GCP_Project. We rely on the folder names and CI/CD workflow to infer the architecture.

##### Elements
- Code push → CI/CD.
- CI/CD → Deployment to Composer & Dataproc.
- Composer → Spark jobs.
- Spark → Data storage.
- CI/CD pipeline (e.g., blue).
- Orchestration layer (e.g., green).
- Compute layer (e.g., orange).
- Storage (e.g., grey).
- Config (e.g., purple).

##### Detailed capture
- Don’t just lump “GCP”—break it down into Composer, Dataproc, GCS/BigQuery.
- If the CI/CD YAML creates or tears down infrastructure, note that (e.g., Terraform or gcloud commands).
- Retries, schedules, or sensor tasks in your Airflow DAG if relevant.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% GitHub and Configuration
    subgraph "GitHub Repository"
        GitHubRepo["GitHub Repo"]:::external
        README["README.md"]:::external
        ConfigRoot["variables/"]:::config
        ConfigDev["dev/variables.json"]:::config
        ConfigProd["prod/variables.json"]:::config
        GitHubRepo --> README
        GitHubRepo --> ConfigRoot
        ConfigRoot --> ConfigDev
        ConfigRoot --> ConfigProd
    end

    %% CI/CD Pipeline
    CI_CD["GitHub Actions CI/CD"]:::cicd
    GitHubRepo --> CI_CD
    ConfigRoot -.-> CI_CD

    %% GCP Environment
    subgraph "GCP Environment"
        subgraph "Cloud Composer"
            Composer["Cloud Composer"]:::orchestration
            Airflow["airflow_job.py"]:::orchestration
        end
        subgraph "Dataproc Cluster"
            Dataproc["Dataproc"]:::compute
            SparkJob["spark_transformation_job.py"]:::compute
        end
        Storage["GCS / BigQuery"]:::storage
    end

    %% Data Consumers
    Consumers["Reports & Dashboards"]:::external

    %% Relationships
    CI_CD -->|"Deploy DAGs"| Composer
    CI_CD -->|"Deploy Spark Job"| Dataproc
    Composer -->|"orchestrates"| Airflow
    Airflow -->|"triggers"| Dataproc
    Dataproc -->|"executes"| SparkJob
    SparkJob -->|"reads/writes"| Storage
    Storage -->|"queried by"| Consumers

    %% Click Events
    click CI_CD "https://github.com/a-vijai/gcp_project/blob/main/.github/workflows/ci-cd.yaml"
    click Airflow "https://github.com/a-vijai/gcp_project/blob/main/airflow_job/airflow_job.py"
    click SparkJob "https://github.com/a-vijai/gcp_project/blob/main/spark_job/spark_transformation_job.py"
    click ConfigDev "https://github.com/a-vijai/gcp_project/blob/main/variables/dev/variables.json"
    click ConfigProd "https://github.com/a-vijai/gcp_project/blob/main/variables/prod/variables.json"
    click ConfigRoot "https://github.com/a-vijai/gcp_project/tree/main/variables/"
    click README "https://github.com/a-vijai/gcp_project/blob/main/README.md"

    %% Styles
    classDef cicd fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef orchestration fill:#90EE90,stroke:#333,stroke-width:1px
    classDef compute fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef config fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#333,stroke-width:1px
```
