xiaoxigua0503/core-banking-bronze-pipeline

How xiaoxigua0503/core-banking-bronze-pipeline works

This is a PostgreSQL-to-BigQuery Bronze ETL pipeline. Its principal workflow is CLI selection → dataset/schema preparation → staging reset → batched PostgreSQL extraction and BigQuery append → merge into core tables. The README documents the SQL scripts’ roles; their contents were not sampled, so SQL relationships below rely on that documentation. Configuration and credentials are represented through the external services rather than as separate components.

Main entry points: main.py Pipeline runner · 01_create_tables.sql Create table SQL · load.py Batch loader · 02_merge_tables.sql Merge SQL

Components 15

Pipeline orchestration

Pipeline runner main.py
SQL file runner main.py
Logging setup etl/logger.py

PostgreSQL extraction

Table batch iterator etl/extract.py
Server-side extraction etl/extract.py

BigQuery loading and merge

Create dataset SQL sql/00_create_dataset.sql
Create table SQL sql/01_create_tables.sql
Staging reset main.py
Batch loader etl/load.py
BigQuery staging tables
Merge SQL sql/02_merge_tables.sql
BigQuery core tables

Other components

CLI operator
PostgreSQL
BigQuery service

Connections 20

CLI operator → Pipeline runner: runs README.md
Pipeline runner → Logging setup: configures logging main.py
Pipeline runner → SQL file runner: calls main.py
SQL file runner → Create dataset SQL: reads main.py
SQL file runner → Create table SQL: reads main.py
SQL file runner → Merge SQL: reads main.py
Pipeline runner → Staging reset: truncates main.py
Pipeline runner → Table batch iterator: iterates batches main.py
Table batch iterator → Server-side extraction: yields batches etl/extract.py
Server-side extraction → PostgreSQL: reads batches etl/extract.py
Pipeline runner → Batch loader: loads batches main.py
Batch loader → BigQuery staging tables: appends rows etl/load.py
Staging reset → BigQuery staging tables: truncates tables main.py
SQL file runner → BigQuery service: executes queries main.py
Batch loader → BigQuery service: submits load jobs etl/load.py
Create dataset SQL → BigQuery service: creates dataset README.md
Create table SQL → BigQuery staging tables: creates staging README.md
Create table SQL → BigQuery core tables: creates tables README.md
Merge SQL → BigQuery staging tables: reads staged rows README.md
Merge SQL → BigQuery core tables: merges rows README.md

Drawn by GitDiagram on November 12, 2026 from the repository's file tree, README and source files. Read as Markdown ·