# xiaoxigua0503/core-banking-bronze-pipeline architecture

> Architecture diagram of the GitHub repository xiaoxigua0503/core-banking-bronze-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xiaoxigua0503/core-banking-bronze-pipeline
- Repository: https://github.com/xiaoxigua0503/core-banking-bronze-pipeline

## Overview

This is a PostgreSQL-to-BigQuery Bronze ETL pipeline. Its principal workflow is CLI selection → dataset/schema preparation → staging reset → batched PostgreSQL extraction and BigQuery append → merge into core tables. The README documents the SQL scripts’ roles; their contents were not sampled, so SQL relationships below rely on that documentation. Configuration and credentials are represented through the external services rather than as separate components.

Main entry points: [`main.py`](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py) (Pipeline runner), [`sql/01_create_tables.sql`](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/01_create_tables.sql) (Create table SQL), [`etl/load.py`](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/load.py) (Batch loader), [`sql/02_merge_tables.sql`](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/02_merge_tables.sql) (Merge SQL)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_runner["Pipeline runner<br/>[main.py]"]
  node_sql_runner["SQL file runner<br/>[main.py]"]
  node_logging["Logging setup<br/>[logger.py]"]
end

subgraph group_extraction["PostgreSQL extraction"]
  node_extract_tables["Table batch iterator<br/>[extract.py]"]
  node_extract_table["Server-side extraction<br/>[extract.py]"]
end

subgraph group_bigquery["BigQuery loading and merge"]
  node_dataset_sql["Create dataset SQL"]
  node_schema_sql["Create table SQL"]
  node_staging_reset["Staging reset<br/>[main.py]"]
  node_batch_loader["Batch loader<br/>[load.py]"]
  node_staging_tables[("BigQuery staging tables")]
  node_merge_sql["Merge SQL"]
  node_core_tables[("BigQuery core tables")]
end

node_cli_user(("CLI operator"))
node_postgres[("PostgreSQL")]
node_bigquery_service[("BigQuery service")]

node_cli_user -->|"runs"| node_runner
node_runner -->|"configures logging"| node_logging
node_runner -->|"calls"| node_sql_runner
node_sql_runner -->|"reads"| node_dataset_sql
node_sql_runner -->|"reads"| node_schema_sql
node_sql_runner -->|"reads"| node_merge_sql
node_runner -->|"truncates"| node_staging_reset
node_runner -->|"iterates batches"| node_extract_tables
node_extract_tables -->|"yields batches"| node_extract_table
node_extract_table -->|"reads batches"| node_postgres
node_runner -->|"loads batches"| node_batch_loader
node_batch_loader -->|"appends rows"| node_staging_tables
node_staging_reset -->|"truncates tables"| node_staging_tables
node_sql_runner -->|"executes queries"| node_bigquery_service
node_batch_loader -->|"submits load jobs"| node_bigquery_service
node_dataset_sql -->|"creates dataset"| node_bigquery_service
node_schema_sql -->|"creates staging"| node_staging_tables
node_schema_sql -->|"creates tables"| node_core_tables
node_merge_sql -->|"reads staged rows"| node_staging_tables
node_merge_sql -->|"merges rows"| node_core_tables

click node_runner "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py"
click node_sql_runner "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py"
click node_logging "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/logger.py"
click node_extract_tables "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/extract.py"
click node_extract_table "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/extract.py"
click node_dataset_sql "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/00_create_dataset.sql"
click node_schema_sql "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/01_create_tables.sql"
click node_staging_reset "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py"
click node_batch_loader "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/load.py"
click node_merge_sql "https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/02_merge_tables.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_sql_runner,node_logging toneBlue
class node_extract_tables,node_extract_table,node_postgres,node_bigquery_service toneAmber
class node_dataset_sql,node_schema_sql,node_staging_reset,node_batch_loader,node_staging_tables,node_merge_sql,node_core_tables toneMint
class node_cli_user toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline runner](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py): `main.py`
- [SQL file runner](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py): `main.py`
- [Logging setup](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/logger.py): `etl/logger.py`

### PostgreSQL extraction

- [Table batch iterator](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/extract.py): `etl/extract.py`
- [Server-side extraction](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/extract.py): `etl/extract.py`

### BigQuery loading and merge

- [Create dataset SQL](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/00_create_dataset.sql): `sql/00_create_dataset.sql`
- [Create table SQL](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/01_create_tables.sql): `sql/01_create_tables.sql`
- [Staging reset](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/main.py): `main.py`
- [Batch loader](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/etl/load.py): `etl/load.py`
- **BigQuery staging tables**
- [Merge SQL](https://github.com/xiaoxigua0503/core-banking-bronze-pipeline/blob/main/sql/02_merge_tables.sql): `sql/02_merge_tables.sql`
- **BigQuery core tables**

### Other components

- **CLI operator**
- **PostgreSQL**
- **BigQuery service**

## Connections

- CLI operator → Pipeline runner: runs (evidence: `README.md`)
- Pipeline runner → Logging setup: configures logging (evidence: `main.py`)
- Pipeline runner → SQL file runner: calls (evidence: `main.py`)
- SQL file runner → Create dataset SQL: reads (evidence: `main.py`)
- SQL file runner → Create table SQL: reads (evidence: `main.py`)
- SQL file runner → Merge SQL: reads (evidence: `main.py`)
- Pipeline runner → Staging reset: truncates (evidence: `main.py`)
- Pipeline runner → Table batch iterator: iterates batches (evidence: `main.py`)
- Table batch iterator → Server-side extraction: yields batches (evidence: `etl/extract.py`)
- Server-side extraction → PostgreSQL: reads batches (evidence: `etl/extract.py`)
- Pipeline runner → Batch loader: loads batches (evidence: `main.py`)
- Batch loader → BigQuery staging tables: appends rows (evidence: `etl/load.py`)
- Staging reset → BigQuery staging tables: truncates tables (evidence: `main.py`)
- SQL file runner → BigQuery service: executes queries (evidence: `main.py`)
- Batch loader → BigQuery service: submits load jobs (evidence: `etl/load.py`)
- Create dataset SQL → BigQuery service: creates dataset (evidence: `README.md`)
- Create table SQL → BigQuery staging tables: creates staging (evidence: `README.md`)
- Create table SQL → BigQuery core tables: creates tables (evidence: `README.md`)
- Merge SQL → BigQuery staging tables: reads staged rows (evidence: `README.md`)
- Merge SQL → BigQuery core tables: merges rows (evidence: `README.md`)