How envoamr/crypto-transactions works
Project’s Purpose and High-Level Components
• This project is built to ingest cryptocurrency market data (from Kaggle) and on-chain transaction data (via BigQuery), transform the data using PySpark, and provide an analysis dashboard (Looker Studio). Its purpose is to reveal the correlation between trade prices and blockchain transaction activity.
• Key components include:
• Infrastructure provisioning (Terraform)
• Workflow orchestration (Kestra)
• Data ingestion and extraction (Python scripts, BigQuery export)
• Data transformation (PySpark job via Dataproc Serverless)
• Storage components (GCS as data lake, BigQuery as data warehouse)
• Visualization (Looker Studio dashboard)
File Structure and Architecture Elements
• Top-level files:
• terraform/: Contains Terraform configuration files (main.tf and variables.tf) that set up all underlying GCP resources (Service Account, VM, firewall, GCS bucket, BigQuery datasets).
• docker-compose.yml: Used to deploy and run Kestra on a VM created by Terraform.
• src/flows: Contains the YAML definitions for the ETL pipeline flows (e.g., etl-pipeline-trigger.yaml) used by Kestra.
• src/scripts: Contains Python scripts for data ingestion (ingest-market-data.py) and data transformation (pyspark-transform-crypto.py).
• src/jars: Contains necessary connector libraries (e.g., gcs-connector and spark–bigquery connector) used by the PySpark job.
• Architectural clues:
• Separation between data ingestion, transformation, orchestration and presentation.
• Use of Infrastructure as Code (Terraform) to provision and secure resources.
• Batch processing (in daily 3 AM workflow and backfill capability) with additional orchestration via Kestra.
• Use of external services (Kaggle, BigQuery, GCS, Dataproc Serverless, Looker Studio).