How pwissemberg/crypto-pipeline works
This repository combines a Python crypto-price ingestion pipeline with a dbt project for staged, time-aggregated, and KPI models. The main runtime flow configures the API and database, fetches historical or incremental hourly BTC/EUR data, checks for missing hours, and loads it into PostgreSQL, optionally using a CSV archive. The dbt files indicate staging, warehouse, and analytics capabilities, but their SQL was not sampled, so their dependencies are not inferred here.
Main entry points: main.py Pipeline runner · Database.py Database access · CryptoAPI.py Crypto API client · btc_eur_hourly.csv CSV archive
Components 23
Ingestion
- Pipeline runner
src/main.py - Runtime config
src/config.py - Crypto API client
src/CryptoAPI.py
Persistence
- Database access
src/Database.py - PostgreSQL
- CSV archive
data/btc_eur_hourly.csv
dbt Models
- Hourly staging
crypto_dbt/models/1_staging/fct_crypto_hourly.sql - Source declarations
crypto_dbt/models/1_staging/sources.yml - Staging schema
crypto_dbt/models/1_staging/staging.yml - Daily warehouse
crypto_dbt/models/2_warehouse/fct_crypto_daily.sql - Weekly warehouse
crypto_dbt/models/2_warehouse/fct_crypto_weekly.sql - Monthly warehouse
crypto_dbt/models/2_warehouse/fct_crypto_monthly.sql - Yearly warehouse
crypto_dbt/models/2_warehouse/fct_crypto_yearly.sql - Warehouse schema
crypto_dbt/models/2_warehouse/warehouse.yml
dbt Analytics
- Daily KPIs
crypto_dbt/models/3_analytics/daily_kpis.sql - Weekly KPIs
crypto_dbt/models/3_analytics/weekly_kpis.sql - Monthly KPIs
crypto_dbt/models/3_analytics/monthly_kpis.sql - Yearly KPIs
crypto_dbt/models/3_analytics/yearly_kpis.sql - Analytics schema
crypto_dbt/models/3_analytics/analytics.yml - KPI macros
crypto_dbt/macros/kpis.sql - Schema macro
crypto_dbt/macros/schema_name.sql
Other components
- Pipeline operator
- CryptoCompare API
Connections 8
- Pipeline operator → Pipeline runner: runs
src/main.py - Pipeline runner → Runtime config: loads settings
src/main.py - Pipeline runner → Crypto API client: fetches prices
src/main.py - Pipeline runner → Database access: loads records
src/main.py - Crypto API client → CryptoCompare API: requests data
src/CryptoAPI.py - Database access → PostgreSQL: queries and writes
src/Database.py - Database access → CSV archive: reads archive
src/Database.py - Pipeline runner → CSV archive: writes archive
src/main.py