# amidarkcoder/banking-data-warehouse architecture

> Architecture diagram of the GitHub repository amidarkcoder/banking-data-warehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amidarkcoder/banking-data-warehouse
- Repository: https://github.com/amidarkcoder/banking-data-warehouse

## Overview

This repository implements a banking-event data warehouse: generate synthetic customer, account, and transaction events, publish them to Kafka, orchestrate ingestion and transformations with Airflow, load warehouse tables, and run quality checks. The graph preserves the generator’s core modules and the documented-in-code Airflow stages. Source excerpts are limited, so connections to unsampled consumers, warehouse loaders, and dashboard configuration are omitted where their runtime wiring is not evidenced.

Main entry points: [`airflow/dags/ingest_pipeline.py`](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/ingest_pipeline.py) (Airflow scheduler), [`pipelines/common.py`](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/common.py) (Pipeline helpers), [`data_generator/generate_all.py`](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/generate_all.py) (Event generator), [`airflow/dags/warehosue_daily_pipeline.py`](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/warehosue_daily_pipeline.py) (Daily warehouse DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Event generation"]
  node_generator["Event generator<br/>[generate_all.py]"]
  node_generator_config["Generator config<br/>[config.py]"]
  node_faker_customer["Customer faker<br/>[faker_customer.py]"]
  node_faker_account["Account faker<br/>[faker_account.py]"]
  node_faker_transaction["Transaction faker"]
end

subgraph group_streaming["Kafka streaming"]
  node_kafka["Kafka broker"]
  node_customer_consumer["Customer consumer"]
  node_account_consumer["Account consumer"]
  node_transaction_consumer["Transaction consumer"]
end

subgraph group_orchestration["Airflow orchestration"]
  node_airflow["Airflow scheduler<br/>[ingest_pipeline.py]"]
  node_dag_helpers["Task launcher<br/>[_dag_helpers.py]"]
  node_warehouse_dag["Daily warehouse DAG"]
  node_quality_dag["Quality DAG"]
end

subgraph group_warehouse["Warehouse processing"]
  node_reference_db[("Reference tables<br/>[seed_branches.sql]")]
  node_raw_ingest["Raw event ingest"]
  node_pipeline_common["Pipeline helpers<br/>[common.py]"]
  node_customer_transform["Customer transform"]
  node_account_transform["Account transform"]
  node_transaction_transform["Transaction transform"]
  node_scd3["Customer SCD3"]
  node_dimension_loader["Dimension loader<br/>[load_dimensions.py]"]
  node_fact_loader["Fact loader"]
  node_warehouse_db[("Warehouse database")]
end

subgraph group_analytics["Quality and analytics"]
  node_quality_checks["Quality checks<br/>[test_faker_data.py]"]
  node_quality_sql["SQL quality checks"]
  node_monitoring["Pipeline metrics<br/>[pipeline_meric.sql]"]
  node_superset["Superset config<br/>[superset_config.py]"]
end

node_operator(("Pipeline operator"))

node_operator -->|"starts"| node_airflow
node_airflow -->|"launches tasks"| node_dag_helpers
node_airflow -->|"runs"| node_generator
node_airflow -->|"runs"| node_customer_consumer
node_airflow -->|"runs"| node_account_consumer
node_airflow -->|"runs"| node_transaction_consumer
node_airflow -->|"runs"| node_customer_transform
node_airflow -->|"runs"| node_account_transform
node_airflow -->|"runs"| node_transaction_transform
node_airflow -->|"runs"| node_scd3
node_airflow -->|"triggers"| node_warehouse_dag
node_warehouse_dag -->|"runs"| node_dimension_loader
node_warehouse_dag -->|"runs"| node_fact_loader
node_warehouse_dag -->|"triggers"| node_quality_dag
node_quality_dag -->|"runs"| node_quality_checks
node_generator -->|"reads settings"| node_generator_config
node_generator -->|"generates customers"| node_faker_customer
node_generator -->|"generates accounts"| node_faker_account
node_generator -->|"generates transactions"| node_faker_transaction
node_generator -->|"reads reference IDs"| node_reference_db
node_generator -->|"publishes events"| node_kafka
node_raw_ingest -->|"consumes events"| node_kafka
node_raw_ingest -->|"gets connection"| node_pipeline_common
node_pipeline_common -->|"records audit runs"| node_warehouse_db
node_pipeline_common -->|"executes SQL"| node_warehouse_db
node_dimension_loader -->|"uses helpers"| node_pipeline_common
node_fact_loader -->|"uses helpers"| node_pipeline_common
node_customer_transform -->|"uses helpers"| node_pipeline_common
node_account_transform -->|"uses helpers"| node_pipeline_common
node_transaction_transform -->|"uses helpers"| node_pipeline_common
node_scd3 -->|"uses helpers"| node_pipeline_common
node_quality_checks -.->|"checks data"| node_quality_sql
node_monitoring -.->|"queries metrics"| node_warehouse_db
node_superset -.->|"connects to"| node_warehouse_db

click node_airflow "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/ingest_pipeline.py"
click node_dag_helpers "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/_dag_helpers.py"
click node_warehouse_dag "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/warehosue_daily_pipeline.py"
click node_quality_dag "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/data_qulity_pipeline.py"
click node_generator "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/generate_all.py"
click node_generator_config "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/config.py"
click node_faker_customer "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_customer.py"
click node_faker_account "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_account.py"
click node_faker_transaction "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_transaction.py"
click node_reference_db "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/database/seeds/seed_branches.sql"
click node_customer_consumer "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/customer_consumer.py"
click node_account_consumer "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/account_consumer.py"
click node_transaction_consumer "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/transaction_consumers.py"
click node_raw_ingest "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/ingest_raw_event.py"
click node_pipeline_common "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/common.py"
click node_customer_transform "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_customers.py"
click node_account_transform "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_account.py"
click node_transaction_transform "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_transaction.py"
click node_scd3 "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/apply_customer_scd3.py"
click node_dimension_loader "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/load_dimensions.py"
click node_fact_loader "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/load_fact_transactions.py"
click node_warehouse_db "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/database/init/005_create_fact_table.sql"
click node_quality_checks "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/tests/test_faker_data.py"
click node_quality_sql "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/sql/quailty/check_duplicates.sql"
click node_monitoring "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/monitoring/pipeline_meric.sql"
click node_superset "https://github.com/amidarkcoder/banking-data-warehouse/blob/main/superset/superset_config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_generator_config,node_faker_customer,node_faker_account,node_faker_transaction toneBlue
class node_kafka,node_customer_consumer,node_account_consumer,node_transaction_consumer toneAmber
class node_airflow,node_dag_helpers,node_warehouse_dag,node_quality_dag toneMint
class node_reference_db,node_raw_ingest,node_pipeline_common,node_customer_transform,node_account_transform,node_transaction_transform,node_scd3,node_dimension_loader,node_fact_loader,node_warehouse_db toneRose
class node_quality_checks,node_quality_sql,node_monitoring,node_superset,node_operator toneIndigo
```

## Components

### Event generation

- [Event generator](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/generate_all.py): `data_generator/generate_all.py`
- [Generator config](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/config.py): `data_generator/config.py`
- [Customer faker](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_customer.py): `data_generator/faker_customer.py`
- [Account faker](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_account.py): `data_generator/faker_account.py`
- [Transaction faker](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/data_generator/faker_transaction.py): `data_generator/faker_transaction.py`

### Kafka streaming

- **Kafka broker**
- [Customer consumer](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/customer_consumer.py): `kafka/consumers/customer_consumer.py`
- [Account consumer](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/account_consumer.py): `kafka/consumers/account_consumer.py`
- [Transaction consumer](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/kafka/consumers/transaction_consumers.py): `kafka/consumers/transaction_consumers.py`

### Airflow orchestration

- [Airflow scheduler](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/ingest_pipeline.py): `airflow/dags/ingest_pipeline.py`
- [Task launcher](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/_dag_helpers.py): `airflow/dags/_dag_helpers.py`
- [Daily warehouse DAG](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/warehosue_daily_pipeline.py): `airflow/dags/warehosue_daily_pipeline.py`
- [Quality DAG](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/airflow/dags/data_qulity_pipeline.py): `airflow/dags/data_qulity_pipeline.py`

### Warehouse processing

- [Reference tables](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/database/seeds/seed_branches.sql): `database/seeds/seed_branches.sql`
- [Raw event ingest](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/ingest_raw_event.py): `pipelines/ingest_raw_event.py`
- [Pipeline helpers](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/common.py): `pipelines/common.py`
- [Customer transform](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_customers.py): `pipelines/transform_customers.py`
- [Account transform](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_account.py): `pipelines/transform_account.py`
- [Transaction transform](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/transform_transaction.py): `pipelines/transform_transaction.py`
- [Customer SCD3](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/apply_customer_scd3.py): `pipelines/apply_customer_scd3.py`
- [Dimension loader](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/load_dimensions.py): `pipelines/load_dimensions.py`
- [Fact loader](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/pipelines/load_fact_transactions.py): `pipelines/load_fact_transactions.py`
- [Warehouse database](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/database/init/005_create_fact_table.sql): `database/init/005_create_fact_table.sql`

### Quality and analytics

- [Quality checks](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/tests/test_faker_data.py): `tests/test_faker_data.py`
- [SQL quality checks](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/sql/quailty/check_duplicates.sql): `sql/quailty/check_duplicates.sql`
- [Pipeline metrics](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/monitoring/pipeline_meric.sql): `monitoring/pipeline_meric.sql`
- [Superset config](https://github.com/amidarkcoder/banking-data-warehouse/blob/main/superset/superset_config.py): `superset/superset_config.py`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Airflow scheduler: starts
- Airflow scheduler → Task launcher: launches tasks (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Event generator: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Customer consumer: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Account consumer: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Transaction consumer: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Customer transform: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Account transform: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Transaction transform: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Customer SCD3: runs (evidence: `airflow/dags/ingest_pipeline.py`)
- Airflow scheduler → Daily warehouse DAG: triggers (evidence: `airflow/dags/ingest_pipeline.py`)
- Daily warehouse DAG → Dimension loader: runs (evidence: `airflow/dags/warehosue_daily_pipeline.py`)
- Daily warehouse DAG → Fact loader: runs (evidence: `airflow/dags/warehosue_daily_pipeline.py`)
- Daily warehouse DAG → Quality DAG: triggers (evidence: `airflow/dags/warehosue_daily_pipeline.py`)
- Quality DAG → Quality checks: runs (evidence: `airflow/dags/data_qulity_pipeline.py`)
- Event generator → Generator config: reads settings (evidence: `data_generator/generate_all.py`)
- Event generator → Customer faker: generates customers (evidence: `data_generator/generate_all.py`)
- Event generator → Account faker: generates accounts (evidence: `data_generator/generate_all.py`)
- Event generator → Transaction faker: generates transactions (evidence: `data_generator/generate_all.py`)
- Event generator → Reference tables: reads reference IDs (evidence: `data_generator/generate_all.py`)
- Event generator → Kafka broker: publishes events (evidence: `data_generator/generate_all.py`)
- Raw event ingest → Kafka broker: consumes events (evidence: `pipelines/ingest_raw_event.py`)
- Raw event ingest → Pipeline helpers: gets connection (evidence: `pipelines/ingest_raw_event.py`)
- Pipeline helpers → Warehouse database: records audit runs (evidence: `pipelines/common.py`)
- Pipeline helpers → Warehouse database: executes SQL (evidence: `pipelines/common.py`)
- Dimension loader → Pipeline helpers: uses helpers (evidence: `pipelines/load_dimensions.py`)
- Fact loader → Pipeline helpers: uses helpers (evidence: `pipelines/load_fact_transactions.py`)
- Customer transform → Pipeline helpers: uses helpers (evidence: `pipelines/transform_customers.py`)
- Account transform → Pipeline helpers: uses helpers (evidence: `pipelines/transform_account.py`)
- Transaction transform → Pipeline helpers: uses helpers (evidence: `pipelines/transform_transaction.py`)
- Customer SCD3 → Pipeline helpers: uses helpers (evidence: `pipelines/apply_customer_scd3.py`)
- Quality checks → SQL quality checks: checks data
- Pipeline metrics → Warehouse database: queries metrics
- Superset config → Warehouse database: connects to
