# pwissemberg/crypto-pipeline architecture

> Architecture diagram of the GitHub repository pwissemberg/crypto-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pwissemberg/crypto-pipeline
- Repository: https://github.com/pwissemberg/crypto-pipeline

## Overview

This repository combines a Python crypto-price ingestion pipeline with a dbt project for staged, time-aggregated, and KPI models. The main runtime flow configures the API and database, fetches historical or incremental hourly BTC/EUR data, checks for missing hours, and loads it into PostgreSQL, optionally using a CSV archive. The dbt files indicate staging, warehouse, and analytics capabilities, but their SQL was not sampled, so their dependencies are not inferred here.

Main entry points: [`src/main.py`](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/main.py) (Pipeline runner), [`src/Database.py`](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/Database.py) (Database access), [`src/CryptoAPI.py`](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/CryptoAPI.py) (Crypto API client), [`data/btc_eur_hourly.csv`](https://github.com/pwissemberg/crypto-pipeline/blob/main/data/btc_eur_hourly.csv) (CSV archive)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_runner["Pipeline runner<br/>[main.py]"]
  node_config["Runtime config<br/>[config.py]"]
  node_api["Crypto API client<br/>[CryptoAPI.py]"]
end

subgraph group_storage["Persistence"]
  node_database["Database access<br/>[Database.py]"]
  node_postgres[("PostgreSQL")]
  node_archive["CSV archive<br/>[btc_eur_hourly.csv]"]
end

subgraph group_warehouse["dbt Models"]
  node_hourly["Hourly staging"]
  node_sources["Source declarations<br/>[sources.yml]"]
  node_staging_schema["Staging schema<br/>[staging.yml]"]
  node_daily["Daily warehouse"]
  node_weekly["Weekly warehouse"]
  node_monthly["Monthly warehouse"]
  node_yearly["Yearly warehouse"]
  node_warehouse_schema["Warehouse schema<br/>[warehouse.yml]"]
end

subgraph group_analytics["dbt Analytics"]
  node_daily_kpi["Daily KPIs<br/>[daily_kpis.sql]"]
  node_weekly_kpi["Weekly KPIs<br/>[weekly_kpis.sql]"]
  node_monthly_kpi["Monthly KPIs<br/>[monthly_kpis.sql]"]
  node_yearly_kpi["Yearly KPIs<br/>[yearly_kpis.sql]"]
  node_analytics_schema["Analytics schema<br/>[analytics.yml]"]
  node_kpi_macro["KPI macros<br/>[kpis.sql]"]
  node_schema_macro["Schema macro<br/>[schema_name.sql]"]
end

node_operator(("Pipeline operator"))
node_crypto_compare["CryptoCompare API"]

node_operator -->|"runs"| node_runner
node_runner -->|"loads settings"| node_config
node_runner -->|"fetches prices"| node_api
node_runner -->|"loads records"| node_database
node_api -->|"requests data"| node_crypto_compare
node_database -->|"queries and writes"| node_postgres
node_database -->|"reads archive"| node_archive
node_runner -->|"writes archive"| node_archive

click node_runner "https://github.com/pwissemberg/crypto-pipeline/blob/main/src/main.py"
click node_config "https://github.com/pwissemberg/crypto-pipeline/blob/main/src/config.py"
click node_api "https://github.com/pwissemberg/crypto-pipeline/blob/main/src/CryptoAPI.py"
click node_database "https://github.com/pwissemberg/crypto-pipeline/blob/main/src/Database.py"
click node_archive "https://github.com/pwissemberg/crypto-pipeline/blob/main/data/btc_eur_hourly.csv"
click node_hourly "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/fct_crypto_hourly.sql"
click node_sources "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/sources.yml"
click node_staging_schema "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/staging.yml"
click node_daily "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_daily.sql"
click node_weekly "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_weekly.sql"
click node_monthly "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_monthly.sql"
click node_yearly "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_yearly.sql"
click node_warehouse_schema "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/warehouse.yml"
click node_daily_kpi "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/daily_kpis.sql"
click node_weekly_kpi "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/weekly_kpis.sql"
click node_monthly_kpi "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/monthly_kpis.sql"
click node_yearly_kpi "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/yearly_kpis.sql"
click node_analytics_schema "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/analytics.yml"
click node_kpi_macro "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/macros/kpis.sql"
click node_schema_macro "https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/macros/schema_name.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_config,node_api toneBlue
class node_database,node_postgres,node_archive toneAmber
class node_hourly,node_sources,node_staging_schema,node_daily,node_weekly,node_monthly,node_yearly,node_warehouse_schema,node_crypto_compare toneMint
class node_daily_kpi,node_weekly_kpi,node_monthly_kpi,node_yearly_kpi,node_analytics_schema,node_kpi_macro,node_schema_macro toneRose
class node_operator toneIndigo
```

## Components

### Ingestion

- [Pipeline runner](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/main.py): `src/main.py`
- [Runtime config](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/config.py): `src/config.py`
- [Crypto API client](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/CryptoAPI.py): `src/CryptoAPI.py`

### Persistence

- [Database access](https://github.com/pwissemberg/crypto-pipeline/blob/main/src/Database.py): `src/Database.py`
- **PostgreSQL**
- [CSV archive](https://github.com/pwissemberg/crypto-pipeline/blob/main/data/btc_eur_hourly.csv): `data/btc_eur_hourly.csv`

### dbt Models

- [Hourly staging](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/fct_crypto_hourly.sql): `crypto_dbt/models/1_staging/fct_crypto_hourly.sql`
- [Source declarations](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/sources.yml): `crypto_dbt/models/1_staging/sources.yml`
- [Staging schema](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/1_staging/staging.yml): `crypto_dbt/models/1_staging/staging.yml`
- [Daily warehouse](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_daily.sql): `crypto_dbt/models/2_warehouse/fct_crypto_daily.sql`
- [Weekly warehouse](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_weekly.sql): `crypto_dbt/models/2_warehouse/fct_crypto_weekly.sql`
- [Monthly warehouse](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_monthly.sql): `crypto_dbt/models/2_warehouse/fct_crypto_monthly.sql`
- [Yearly warehouse](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/fct_crypto_yearly.sql): `crypto_dbt/models/2_warehouse/fct_crypto_yearly.sql`
- [Warehouse schema](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/2_warehouse/warehouse.yml): `crypto_dbt/models/2_warehouse/warehouse.yml`

### dbt Analytics

- [Daily KPIs](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/daily_kpis.sql): `crypto_dbt/models/3_analytics/daily_kpis.sql`
- [Weekly KPIs](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/weekly_kpis.sql): `crypto_dbt/models/3_analytics/weekly_kpis.sql`
- [Monthly KPIs](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/monthly_kpis.sql): `crypto_dbt/models/3_analytics/monthly_kpis.sql`
- [Yearly KPIs](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/yearly_kpis.sql): `crypto_dbt/models/3_analytics/yearly_kpis.sql`
- [Analytics schema](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/models/3_analytics/analytics.yml): `crypto_dbt/models/3_analytics/analytics.yml`
- [KPI macros](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/macros/kpis.sql): `crypto_dbt/macros/kpis.sql`
- [Schema macro](https://github.com/pwissemberg/crypto-pipeline/blob/main/crypto_dbt/macros/schema_name.sql): `crypto_dbt/macros/schema_name.sql`

### Other components

- **Pipeline operator**
- **CryptoCompare API**

## Connections

- Pipeline operator → Pipeline runner: runs (evidence: `src/main.py`)
- Pipeline runner → Runtime config: loads settings (evidence: `src/main.py`)
- Pipeline runner → Crypto API client: fetches prices (evidence: `src/main.py`)
- Pipeline runner → Database access: loads records (evidence: `src/main.py`)
- Crypto API client → CryptoCompare API: requests data (evidence: `src/CryptoAPI.py`)
- Database access → PostgreSQL: queries and writes (evidence: `src/Database.py`)
- Database access → CSV archive: reads archive (evidence: `src/Database.py`)
- Pipeline runner → CSV archive: writes archive (evidence: `src/main.py`)
