How amidarkcoder/banking-data-warehouse works
This repository implements a banking-event data warehouse: generate synthetic customer, account, and transaction events, publish them to Kafka, orchestrate ingestion and transformations with Airflow, load warehouse tables, and run quality checks. The graph preserves the generator’s core modules and the documented-in-code Airflow stages. Source excerpts are limited, so connections to unsampled consumers, warehouse loaders, and dashboard configuration are omitted where their runtime wiring is not evidenced.
Main entry points: ingest_pipeline.py Airflow scheduler · common.py Pipeline helpers · generate_all.py Event generator · warehosue_daily_pipeline.py Daily warehouse DAG
Components 28
Event generation
- Event generator
data_generator/generate_all.py - Generator config
data_generator/config.py - Customer faker
data_generator/faker_customer.py - Account faker
data_generator/faker_account.py - Transaction faker
data_generator/faker_transaction.py
Kafka streaming
- Kafka broker
- Customer consumer
kafka/consumers/customer_consumer.py - Account consumer
kafka/consumers/account_consumer.py - Transaction consumer
kafka/consumers/transaction_consumers.py
Airflow orchestration
- Airflow scheduler
airflow/dags/ingest_pipeline.py - Task launcher
airflow/dags/_dag_helpers.py - Daily warehouse DAG
airflow/dags/warehosue_daily_pipeline.py - Quality DAG
airflow/dags/data_qulity_pipeline.py
Warehouse processing
- Reference tables
database/seeds/seed_branches.sql - Raw event ingest
pipelines/ingest_raw_event.py - Pipeline helpers
pipelines/common.py - Customer transform
pipelines/transform_customers.py - Account transform
pipelines/transform_account.py - Transaction transform
pipelines/transform_transaction.py - Customer SCD3
pipelines/apply_customer_scd3.py - Dimension loader
pipelines/load_dimensions.py - Fact loader
pipelines/load_fact_transactions.py - Warehouse database
database/init/005_create_fact_table.sql
Quality and analytics
- Quality checks
tests/test_faker_data.py - SQL quality checks
sql/quailty/check_duplicates.sql - Pipeline metrics
monitoring/pipeline_meric.sql - Superset config
superset/superset_config.py
Other components
- Pipeline operator
Connections 34
- Pipeline operator → Airflow scheduler: starts
- Airflow scheduler → Task launcher: launches tasks
airflow/dags/ingest_pipeline.py - Airflow scheduler → Event generator: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Customer consumer: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Account consumer: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Transaction consumer: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Customer transform: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Account transform: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Transaction transform: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Customer SCD3: runs
airflow/dags/ingest_pipeline.py - Airflow scheduler → Daily warehouse DAG: triggers
airflow/dags/ingest_pipeline.py - Daily warehouse DAG → Dimension loader: runs
airflow/dags/warehosue_daily_pipeline.py - Daily warehouse DAG → Fact loader: runs
airflow/dags/warehosue_daily_pipeline.py - Daily warehouse DAG → Quality DAG: triggers
airflow/dags/warehosue_daily_pipeline.py - Quality DAG → Quality checks: runs
airflow/dags/data_qulity_pipeline.py - Event generator → Generator config: reads settings
data_generator/generate_all.py - Event generator → Customer faker: generates customers
data_generator/generate_all.py - Event generator → Account faker: generates accounts
data_generator/generate_all.py - Event generator → Transaction faker: generates transactions
data_generator/generate_all.py - Event generator → Reference tables: reads reference IDs
data_generator/generate_all.py - Event generator → Kafka broker: publishes events
data_generator/generate_all.py - Raw event ingest → Kafka broker: consumes events
pipelines/ingest_raw_event.py - Raw event ingest → Pipeline helpers: gets connection
pipelines/ingest_raw_event.py - Pipeline helpers → Warehouse database: records audit runs
pipelines/common.py - Pipeline helpers → Warehouse database: executes SQL
pipelines/common.py - Dimension loader → Pipeline helpers: uses helpers
pipelines/load_dimensions.py - Fact loader → Pipeline helpers: uses helpers
pipelines/load_fact_transactions.py - Customer transform → Pipeline helpers: uses helpers
pipelines/transform_customers.py - Account transform → Pipeline helpers: uses helpers
pipelines/transform_account.py - Transaction transform → Pipeline helpers: uses helpers
pipelines/transform_transaction.py - Customer SCD3 → Pipeline helpers: uses helpers
pipelines/apply_customer_scd3.py - Quality checks → SQL quality checks: checks data
- Pipeline metrics → Warehouse database: queries metrics
- Superset config → Warehouse database: connects to