How rionurfadilah229/banking-transaction-elt works
This repository is a banking transaction ELT pipeline: synthetic data feeds PostgreSQL, Airflow-triggered Python ingestion loads BigQuery, and dbt builds staging and dimensional analytics models. It supports full and timestamp-watermarked incremental transaction loads, plus documented warehouse quality checks. Source excerpts show the Python load and extraction calls; dbt relationships and testing capabilities are documented in the README, but their SQL implementations were not sampled.
Main entry points: load.py BigQuery Loader · extract.py Python Extraction · marts Dimensional Marts · ddl PostgreSQL Source
Components 13
Source Data
- Synthetic Generator
src/generate_data.py - PostgreSQL Source
sql/ddl
Ingestion
- Extraction DAG
dags/banking_ingestion_dag.py - Python Extraction
src/ingestion/extract.py - Initial Load DAG
dags/banking_initial_load_dag.py - Incremental Load DAG
dags/banking_incremental_load_dag.py - BigQuery Loader
src/ingestion/load.py - PostgreSQL Connection
src/ingestion/db.py
BigQuery Warehouse
- banking_raw
dbt Modeling
- Staging Models
dbt/banking_warehouse/models/staging - Dimensional Marts
dbt/banking_warehouse/models/marts - Data Quality Tests
dbt/banking_warehouse/models/marts/schema.yml
Other components
- Analytics Consumer
Connections 13
- Synthetic Generator → PostgreSQL Source: loads data
README.md - Extraction DAG → Python Extraction: invokes tasks
dags/banking_ingestion_dag.py - Python Extraction → PostgreSQL Connection: gets engine
src/ingestion/extract.py - Python Extraction → PostgreSQL Source: queries tables
src/ingestion/extract.py - Initial Load DAG → BigQuery Loader: invokes tasks
dags/banking_initial_load_dag.py - Incremental Load DAG → BigQuery Loader: invokes task
dags/banking_incremental_load_dag.py - BigQuery Loader → PostgreSQL Connection: gets engine
src/ingestion/load.py - BigQuery Loader → PostgreSQL Source: reads tables
src/ingestion/load.py - BigQuery Loader → banking_raw: loads and appends
src/ingestion/load.py - banking_raw → Staging Models: transforms sources
README.md - Staging Models → Dimensional Marts: feeds models
README.md - Dimensional Marts → Data Quality Tests: validates integrity
README.md - Dimensional Marts → Analytics Consumer: serves analytics
README.md