# beiqisu10/crypto_data_engineering architecture

> Architecture diagram of the GitHub repository beiqisu10/crypto_data_engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/beiqisu10/crypto_data_engineering
- Repository: https://github.com/beiqisu10/crypto_data_engineering

## Overview

#### Purpose and Entry Points
This repository implements a batch crypto analytics pipeline that joins Binance BTC/USDT market data, CoinGecko prices, and Fear & Greed sentiment to evaluate sentiment-versus-return behavior. Primary operational entry points are `Makefile` (`make setup`, infrastructure, Airflow, and dbt commands), Airflow DAGs, and standalone ingestion scripts in `scripts/`.

#### Infrastructure Boundary: GCP Resources
Terraform provisions the cloud storage and warehouse layer consumed by ingestion and transformation workloads: GCS is the raw landing/data-lake boundary, while BigQuery is the modeled analytics boundary. Infrastructure definitions are concentrated in:
- `terraform/main.tf`
- `terraform/providers.tf`
- `terraform/variables.tf`

The runtime requires a GCP project, enabled BigQuery/GCS APIs, and service-account credentials; Airflow uses a `google_cloud_default` connection and dbt uses its BigQuery profile.

#### Orchestration Layer
Airflow runs containerized workflows and establishes ordering between independently deployable ingestion pipelines:
- `airflow/docker-compose.yaml`
- `airflow/dags/binance_ingestion_dag.py`
- `airflow/dags/market_data_dag.py`

`binance_spark_ingestion_v1` is the source-of-truth market-data workflow. The market metadata/sentiment DAG waits on its successful completion through an `ExternalTaskSensor`, preventing downstream sentiment processing against absent or stale price data. DAG tasks retry transient failures twice with five-minute delays and include failure-callback hooks.

#### Ingestion and Raw Data Flow
Python ingestion jobs retrieve external API data and deliver it into the GCS/BigQuery landing path:
- `scripts/ingest_binance.py`
- `scripts/ingest_coingecko.py`
- `scripts/ingest_sentiment.py`

The intended flow is API extraction → raw CSV landing in GCS → Spark/PySpark cleaning and Parquet conversion → BigQuery external or native raw/staging tables. External dependencies are Binance, CoinGecko, and Alternative.me Fear & Greed data; their rate limits and available historical windows constrain current coverage.

#### Transformation and Semantic Modeling
dbt is the warehouse transformation boundary. It converts landing sources into normalized staging models, then produces an analytics-ready fact table:
- `dbt_crypto/models/staging/stg_crypto__btcusdt.sql`
- `dbt_crypto/models/staging/stg_crypto__prices.sql`
- `dbt_crypto/models/staging/stg_crypto__sentiment.sql`
- `dbt_crypto/models/marts/fct_crypto_daily_metrics.sql`

The three staging streams converge by date in `fct_crypto_daily_metrics`, where SQL window logic derives volatility, moving measures, and forward-return-oriented metrics for sentiment analysis.

#### Serving and Warehouse Optimization
The final BigQuery fact model is designed for dashboard consumption, including Looker Studio visualizations of fear/return relationships and price-versus-sentiment trends:
- `dbt_crypto/models/marts/fct_crypto_daily_metrics.sql`
- `dbt_crypto/models/marts/fct_crypto_daily_metrics.yml`

It is partitioned daily on `date_day` to constrain time-range scans and clustered by `sentiment_label` for efficient sentiment-segment queries. dbt schema metadata defines core fact-table integrity expectations, including unique and non-null daily keys.

#### Runtime Topology
Airflow runs in Docker via the repository-specific image and Compose environment:
- `airflow/Dockerfile`
- `airflow/docker-compose.yaml`
- `pyproject.toml`

`main.py` is the top-level Python module, while Airflow is the primary scheduled execution surface. dbt execution is intentionally run from the Airflow container so warehouse permissions and dependencies align with orchestration runtime configuration.

Main entry points: [`airflow/dags/market_data_dag.py`](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/market_data_dag.py) (Metadata & sentiment DAG), [`airflow/docker-compose.yaml`](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/docker-compose.yaml) (Airflow Compose runtime), [`dbt_crypto/models/marts/fct_crypto_daily_metrics.sql`](https://github.com/beiqisu10/crypto_data_engineering/blob/main/dbt_crypto/models/marts/fct_crypto_daily_metrics.sql) (Daily crypto metrics fact), [`airflow/dags/binance_ingestion_dag.py`](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/binance_ingestion_dag.py) (Binance ingestion DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infra["GCP Infrastructure"]
  node_terraform{{"Terraform resources<br/>infrastructure<br/>[main.tf]"}}
  node_gcs[("GCS raw landing<br/>data lake")]
  node_bigquery[("BigQuery warehouse")]
end

subgraph group_orchestration["Orchestration Runtime"]
  node_compose["Airflow Compose runtime<br/>container runtime"]
  node_airflow_image["Airflow image<br/>container build"]
  node_binance_dag["Binance ingestion DAG<br/>Airflow DAG"]
  node_metadata_dag["Metadata &amp; sentiment DAG<br/>Airflow DAG<br/>[market_data_dag.py]"]
end

subgraph group_ingestion["Ingestion Pipeline"]
  node_binance_api(("Binance API<br/>external API"))
  node_coingecko_api(("CoinGecko API<br/>external API"))
  node_fear_greed_api(("Fear &amp; Greed API<br/>external API"))
  node_binance_ingest["Binance extractor<br/>Python ingestion<br/>[ingest_binance.py]"]
  node_coingecko_ingest["CoinGecko extractor<br/>Python ingestion"]
  node_sentiment_ingest["Sentiment extractor<br/>Python ingestion"]
  node_spark_clean["Spark clean &amp; Parquet<br/>processing stage"]
end

subgraph group_warehouse["Warehouse Modeling"]
  node_dbt_staging["dbt staging models<br/>dbt models"]
  node_daily_fact["Daily crypto metrics fact<br/>dbt mart"]
end

node_looker["Looker Studio dashboards<br/>BI serving"]

node_terraform -->|"provisions"| node_gcs
node_terraform -->|"provisions"| node_bigquery
node_airflow_image -->|"builds runtime for"| node_compose
node_compose -->|"runs"| node_binance_dag
node_compose -->|"runs"| node_metadata_dag
node_metadata_dag -.->|"waits for success"| node_binance_dag
node_binance_dag -->|"orchestrates"| node_binance_ingest
node_metadata_dag -->|"orchestrates"| node_coingecko_ingest
node_metadata_dag -->|"orchestrates"| node_sentiment_ingest
node_binance_api -->|"market data"| node_binance_ingest
node_coingecko_api -->|"prices"| node_coingecko_ingest
node_fear_greed_api -->|"sentiment"| node_sentiment_ingest
node_binance_ingest -->|"lands raw CSV"| node_gcs
node_coingecko_ingest -->|"lands raw CSV"| node_gcs
node_sentiment_ingest -->|"lands raw CSV"| node_gcs
node_gcs -->|"raw CSV"| node_spark_clean
node_spark_clean -->|"raw/staging data"| node_bigquery
node_bigquery -->|"landing sources"| node_dbt_staging
node_dbt_staging -->|"three dated streams"| node_daily_fact
node_daily_fact -->|"materializes partitioned fact"| node_bigquery
node_daily_fact -->|"serves analytics"| node_looker

click node_terraform "https://github.com/beiqisu10/crypto_data_engineering/blob/main/terraform/main.tf"
click node_compose "https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/docker-compose.yaml"
click node_airflow_image "https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/Dockerfile"
click node_binance_dag "https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/binance_ingestion_dag.py"
click node_metadata_dag "https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/market_data_dag.py"
click node_binance_ingest "https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_binance.py"
click node_coingecko_ingest "https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_coingecko.py"
click node_sentiment_ingest "https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_sentiment.py"
click node_dbt_staging "https://github.com/beiqisu10/crypto_data_engineering/blob/main/dbt_crypto/models/staging/stg_crypto__btcusdt.sql"
click node_daily_fact "https://github.com/beiqisu10/crypto_data_engineering/blob/main/dbt_crypto/models/marts/fct_crypto_daily_metrics.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_terraform,node_gcs,node_bigquery toneBlue
class node_compose,node_airflow_image,node_binance_dag,node_metadata_dag toneAmber
class node_binance_api,node_coingecko_api,node_fear_greed_api,node_binance_ingest,node_coingecko_ingest,node_sentiment_ingest,node_spark_clean toneMint
class node_dbt_staging,node_daily_fact toneRose
class node_looker toneNeutral
```

## Components

### GCP Infrastructure

- [Terraform resources](https://github.com/beiqisu10/crypto_data_engineering/blob/main/terraform/main.tf): infrastructure · Provisions GCS and BigQuery boundaries. · `terraform/main.tf`
- **GCS raw landing**: data lake · Raw CSV landing boundary.
- **BigQuery warehouse**: warehouse · Raw, staging, and analytics tables.

### Orchestration Runtime

- [Airflow Compose runtime](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/docker-compose.yaml): container runtime · `airflow/docker-compose.yaml`
- [Airflow image](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/Dockerfile): container build · `airflow/Dockerfile`
- [Binance ingestion DAG](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/binance_ingestion_dag.py): Airflow DAG · Source-of-truth market-data workflow. · `airflow/dags/binance_ingestion_dag.py`
- [Metadata & sentiment DAG](https://github.com/beiqisu10/crypto_data_engineering/blob/main/airflow/dags/market_data_dag.py): Airflow DAG · Waits for successful market-data completion. · `airflow/dags/market_data_dag.py`

### Ingestion Pipeline

- **Binance API**: external API
- **CoinGecko API**: external API
- **Fear & Greed API**: external API
- [Binance extractor](https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_binance.py): Python ingestion · `scripts/ingest_binance.py`
- [CoinGecko extractor](https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_coingecko.py): Python ingestion · `scripts/ingest_coingecko.py`
- [Sentiment extractor](https://github.com/beiqisu10/crypto_data_engineering/blob/main/scripts/ingest_sentiment.py): Python ingestion · `scripts/ingest_sentiment.py`
- **Spark clean & Parquet**: processing stage · Cleans landed CSV and converts it to Parquet.

### Warehouse Modeling

- [dbt staging models](https://github.com/beiqisu10/crypto_data_engineering/blob/main/dbt_crypto/models/staging/stg_crypto__btcusdt.sql): dbt models · Normalizes BTCUSDT, prices, and sentiment streams. · `dbt_crypto/models/staging/stg_crypto__btcusdt.sql`
- [Daily crypto metrics fact](https://github.com/beiqisu10/crypto_data_engineering/blob/main/dbt_crypto/models/marts/fct_crypto_daily_metrics.sql): dbt mart · Joins streams by date and derives volatility and forward returns. · `dbt_crypto/models/marts/fct_crypto_daily_metrics.sql`

### Other components

- **Looker Studio dashboards**: BI serving · Analyzes fear/return and price/sentiment trends.

## Connections

- Terraform resources → GCS raw landing: provisions
- Terraform resources → BigQuery warehouse: provisions
- Airflow image → Airflow Compose runtime: builds runtime for
- Airflow Compose runtime → Binance ingestion DAG: runs
- Airflow Compose runtime → Metadata & sentiment DAG: runs
- Metadata & sentiment DAG → Binance ingestion DAG: waits for success · ExternalTaskSensor dependency.
- Binance ingestion DAG → Binance extractor: orchestrates
- Metadata & sentiment DAG → CoinGecko extractor: orchestrates
- Metadata & sentiment DAG → Sentiment extractor: orchestrates
- Binance API → Binance extractor: market data
- CoinGecko API → CoinGecko extractor: prices
- Fear & Greed API → Sentiment extractor: sentiment
- Binance extractor → GCS raw landing: lands raw CSV
- CoinGecko extractor → GCS raw landing: lands raw CSV
- Sentiment extractor → GCS raw landing: lands raw CSV
- GCS raw landing → Spark clean & Parquet: raw CSV
- Spark clean & Parquet → BigQuery warehouse: raw/staging data
- BigQuery warehouse → dbt staging models: landing sources
- dbt staging models → Daily crypto metrics fact: three dated streams
- Daily crypto metrics fact → BigQuery warehouse: materializes partitioned fact
- Daily crypto metrics fact → Looker Studio dashboards: serves analytics