How a-vijai/gcp_project works
Project type and purpose
• This is a GCP-based data-pipeline project. It consists of:
• An Airflow job (airflow_job/airflow_job.py) for orchestration.
• A Spark job (spark_job/spark_transformation_job.py) for data transformation.
• Environment-specific configuration files (variables/dev/variables.json and variables/prod/variables.json).
• A CI/CD pipeline defined in .github/workflows/ci-cd.yaml to deploy Airflow DAGs and Spark jobs to GCP.
File structure
• Top-level:
• .github/workflows/ci-cd.yaml – CI/CD workflow.
• airflow_job/ – contains the Airflow DAG or operator code.
• spark_job/ – contains the Spark transformation logic.
• variables/dev and variables/prod – JSON config files holding parameters for each environment.
• There are no separate “frontend” or “backend” folders—this is purely a data orchestration and processing project.
• CI/CD files indicate automated deployment of jobs to GCP.
README for additional insights
• The README is minimal; it only names the project as GCP_Project. We rely on the folder names and CI/CD workflow to infer the architecture.
Elements
• Code push → CI/CD.
• CI/CD → Deployment to Composer & Dataproc.
• Composer → Spark jobs.
• Spark → Data storage.
• CI/CD pipeline (e.g., blue).
• Orchestration layer (e.g., green).
• Compute layer (e.g., orange).
• Storage (e.g., grey).
• Config (e.g., purple).
Detailed capture
• Don’t just lump “GCP”—break it down into Composer, Dataproc, GCS/BigQuery.
• If the CI/CD YAML creates or tears down infrastructure, note that (e.g., Terraform or gcloud commands).
• Retries, schedules, or sensor tasks in your Airflow DAG if relevant.