# rionurfadilah229/banking-transaction-elt architecture

> Architecture diagram of the GitHub repository rionurfadilah229/banking-transaction-elt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rionurfadilah229/banking-transaction-elt
- Repository: https://github.com/rionurfadilah229/banking-transaction-elt

## Overview

This repository is a banking transaction ELT pipeline: synthetic data feeds PostgreSQL, Airflow-triggered Python ingestion loads BigQuery, and dbt builds staging and dimensional analytics models. It supports full and timestamp-watermarked incremental transaction loads, plus documented warehouse quality checks. Source excerpts show the Python load and extraction calls; dbt relationships and testing capabilities are documented in the README, but their SQL implementations were not sampled.

Main entry points: [`src/ingestion/load.py`](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/load.py) (BigQuery Loader), [`src/ingestion/extract.py`](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/extract.py) (Python Extraction), [`dbt/banking_warehouse/models/marts`](https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/dbt/banking_warehouse/models/marts) (Dimensional Marts), [`sql/ddl`](https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/sql/ddl) (PostgreSQL Source)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_source["Source Data"]
  node_generator["Synthetic Generator<br/>[generate_data.py]"]
  node_postgres[("PostgreSQL Source")]
end

subgraph group_orchestration["Ingestion"]
  node_ingestion_dag["Extraction DAG"]
  node_extract["Python Extraction<br/>[extract.py]"]
  node_initial_dag["Initial Load DAG"]
  node_incremental_dag["Incremental Load DAG"]
  node_loader["BigQuery Loader<br/>[load.py]"]
  node_db_connection["PostgreSQL Connection<br/>[db.py]"]
end

subgraph group_warehouse["BigQuery Warehouse"]
  node_bigquery_raw[("banking_raw")]
end

subgraph group_modeling["dbt Modeling"]
  node_staging["Staging Models"]
  node_marts["Dimensional Marts"]
  node_quality["Data Quality Tests<br/>[schema.yml]"]
end

node_analytics(("Analytics Consumer"))

node_generator -->|"loads data"| node_postgres
node_ingestion_dag -->|"invokes tasks"| node_extract
node_extract -->|"gets engine"| node_db_connection
node_extract -->|"queries tables"| node_postgres
node_initial_dag -->|"invokes tasks"| node_loader
node_incremental_dag -->|"invokes task"| node_loader
node_loader -->|"gets engine"| node_db_connection
node_loader -->|"reads tables"| node_postgres
node_loader -->|"loads and appends"| node_bigquery_raw
node_bigquery_raw -->|"transforms sources"| node_staging
node_staging -->|"feeds models"| node_marts
node_marts -->|"validates integrity"| node_quality
node_marts -->|"serves analytics"| node_analytics

click node_generator "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/generate_data.py"
click node_postgres "https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/sql/ddl"
click node_ingestion_dag "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_ingestion_dag.py"
click node_extract "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/extract.py"
click node_initial_dag "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_initial_load_dag.py"
click node_incremental_dag "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_incremental_load_dag.py"
click node_loader "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/load.py"
click node_db_connection "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/db.py"
click node_staging "https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/dbt/banking_warehouse/models/staging"
click node_marts "https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/dbt/banking_warehouse/models/marts"
click node_quality "https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dbt/banking_warehouse/models/marts/schema.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_postgres toneBlue
class node_ingestion_dag,node_extract,node_initial_dag,node_incremental_dag,node_loader,node_db_connection toneAmber
class node_bigquery_raw toneMint
class node_staging,node_marts,node_quality toneRose
class node_analytics toneIndigo
```

## Components

### Source Data

- [Synthetic Generator](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/generate_data.py): `src/generate_data.py`
- [PostgreSQL Source](https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/sql/ddl): `sql/ddl`

### Ingestion

- [Extraction DAG](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_ingestion_dag.py): `dags/banking_ingestion_dag.py`
- [Python Extraction](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/extract.py): `src/ingestion/extract.py`
- [Initial Load DAG](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_initial_load_dag.py): `dags/banking_initial_load_dag.py`
- [Incremental Load DAG](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dags/banking_incremental_load_dag.py): `dags/banking_incremental_load_dag.py`
- [BigQuery Loader](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/load.py): `src/ingestion/load.py`
- [PostgreSQL Connection](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/src/ingestion/db.py): `src/ingestion/db.py`

### BigQuery Warehouse

- **banking_raw**

### dbt Modeling

- [Staging Models](https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/dbt/banking_warehouse/models/staging): `dbt/banking_warehouse/models/staging`
- [Dimensional Marts](https://github.com/rionurfadilah229/banking-transaction-elt/tree/main/dbt/banking_warehouse/models/marts): `dbt/banking_warehouse/models/marts`
- [Data Quality Tests](https://github.com/rionurfadilah229/banking-transaction-elt/blob/main/dbt/banking_warehouse/models/marts/schema.yml): `dbt/banking_warehouse/models/marts/schema.yml`

### Other components

- **Analytics Consumer**

## Connections

- Synthetic Generator → PostgreSQL Source: loads data (evidence: `README.md`)
- Extraction DAG → Python Extraction: invokes tasks (evidence: `dags/banking_ingestion_dag.py`)
- Python Extraction → PostgreSQL Connection: gets engine (evidence: `src/ingestion/extract.py`)
- Python Extraction → PostgreSQL Source: queries tables (evidence: `src/ingestion/extract.py`)
- Initial Load DAG → BigQuery Loader: invokes tasks (evidence: `dags/banking_initial_load_dag.py`)
- Incremental Load DAG → BigQuery Loader: invokes task (evidence: `dags/banking_incremental_load_dag.py`)
- BigQuery Loader → PostgreSQL Connection: gets engine (evidence: `src/ingestion/load.py`)
- BigQuery Loader → PostgreSQL Source: reads tables (evidence: `src/ingestion/load.py`)
- BigQuery Loader → banking_raw: loads and appends (evidence: `src/ingestion/load.py`)
- banking_raw → Staging Models: transforms sources (evidence: `README.md`)
- Staging Models → Dimensional Marts: feeds models (evidence: `README.md`)
- Dimensional Marts → Data Quality Tests: validates integrity (evidence: `README.md`)
- Dimensional Marts → Analytics Consumer: serves analytics (evidence: `README.md`)
