# envoamr/crypto-transactions architecture

> Architecture diagram of the GitHub repository envoamr/crypto-transactions, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/envoamr/crypto-transactions
- Repository: https://github.com/envoamr/crypto-transactions

## Overview

##### Project’s Purpose and High-Level Components
- This project is built to ingest cryptocurrency market data (from Kaggle) and on-chain transaction data (via BigQuery), transform the data using PySpark, and provide an analysis dashboard (Looker Studio). Its purpose is to reveal the correlation between trade prices and blockchain transaction activity.
- Key components include:
- Infrastructure provisioning (Terraform)
- Workflow orchestration (Kestra)
- Data ingestion and extraction (Python scripts, BigQuery export)
- Data transformation (PySpark job via Dataproc Serverless)
- Storage components (GCS as data lake, BigQuery as data warehouse)
- Visualization (Looker Studio dashboard)

##### File Structure and Architecture Elements
- Top-level files:
- terraform/: Contains Terraform configuration files (main.tf and variables.tf) that set up all underlying GCP resources (Service Account, VM, firewall, GCS bucket, BigQuery datasets).
- docker-compose.yml: Used to deploy and run Kestra on a VM created by Terraform.
- src/flows: Contains the YAML definitions for the ETL pipeline flows (e.g., etl-pipeline-trigger.yaml) used by Kestra.
- src/scripts: Contains Python scripts for data ingestion (ingest-market-data.py) and data transformation (pyspark-transform-crypto.py).
- src/jars: Contains necessary connector libraries (e.g., gcs-connector and spark–bigquery connector) used by the PySpark job.
- Architectural clues:
- Separation between data ingestion, transformation, orchestration and presentation.
- Use of Infrastructure as Code (Terraform) to provision and secure resources.
- Batch processing (in daily 3 AM workflow and backfill capability) with additional orchestration via Kestra.
- Use of external services (Kaggle, BigQuery, GCS, Dataproc Serverless, Looker Studio).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Infrastructure Provisioning
    subgraph "Infrastructure Provisioning"
        T["Terraform (provisions VM, GCS, BigQuery, IAM, Firewall)"]:::infra
    end

    %% Workflow Orchestration Layer
    subgraph "Workflow Orchestration"
        K["Kestra (Docker Compose on VM)"]:::orchestration
        F["ETL Pipeline Flows (YAML Definitions)"]:::orchestration
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        P["Market Data Extraction (Python Script)"]:::ingestion
        BQExp["On-chain Data Export (BigQuery)"]:::ingestion
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        GCS["GCS Bucket (Data Lake)"]:::storage
        BQ["BigQuery (Data Warehouse)"]:::storage
    end

    %% Data Transformation Layer
    subgraph "Data Transformation"
        Spark["PySpark Job (Dataproc Serverless)"]:::transformation
        Script["Transformation Script (PySpark)"]:::transformation
        Jars["Connector Libraries (JARs)"]:::transformation
    end

    %% Visualization Layer
    subgraph "Visualization"
        Looker["Looker Studio Dashboard"]:::visualization
    end

    %% Connections between layers
    T -->|"provisions"| K
    T -->|"provisions"| GCS
    T -->|"provisions"| BQ

    K -->|"orchestrates"| F
    K -->|"triggers"| P
    K -->|"initiates"| Spark

    P -->|"sends raw data"| GCS
    BQExp -->|"sends exported data"| GCS

    GCS -->|"feeds raw data"| Spark
    Script -->|"executes within"| Spark
    Jars -->|"supplies connectors to"| Spark

    Spark -->|"transforms & loads data"| BQ

    BQ -->|"provides data to"| Looker

    %% Click Events for Component Mapping
    click T "https://github.com/envoamr/crypto-transactions/blob/main/terraform/main.tf"
    click T "https://github.com/envoamr/crypto-transactions/blob/main/terraform/variables.tf"
    click K "https://github.com/envoamr/crypto-transactions/blob/main/docker-compose.yml"
    click F "https://github.com/envoamr/crypto-transactions/blob/main/src/flows/etl-pipeline-trigger.yaml"
    click P "https://github.com/envoamr/crypto-transactions/blob/main/src/scripts/ingest-market-data.py"
    click Script "https://github.com/envoamr/crypto-transactions/blob/main/src/scripts/pyspark-transform-crypto.py"
    click Jars "https://github.com/envoamr/crypto-transactions/tree/main/src/jars/"

    %% Styling
    classDef infra fill:#fdd,stroke:#a33,stroke-width:2px;
    classDef orchestration fill:#ddf,stroke:#337,stroke-width:2px;
    classDef ingestion fill:#dfd,stroke:#373,stroke-width:2px;
    classDef storage fill:#ffd,stroke:#993,stroke-width:2px;
    classDef transformation fill:#def,stroke:#359,stroke-width:2px;
    classDef visualization fill:#fdf,stroke:#935,stroke-width:2px;
```
