How omarfarooq908/fintech-lakehouse-pipeline works
This is a synthetic fintech data lakehouse: configurable fintech records feed partitioned Parquet, DuckDB raw views, and dbt models for curated analytics and risk. The principal workflow is batch generation and ingestion followed by staging, intermediate transformations, and marts; Airflow is documented as the orchestrator, but its source is absent from the tree. The graph preserves documented outputs and reliability subsystems, while limiting internal transformation links where source excerpts do not show model dependencies.
Main entry points: core Core dimensions · parquet_lakehouse.py Batch writer · intermediate Intermediate models · api_server.py Synthetic API
Components 19
Synthetic data
- Synthetic API
simulator/api_server.py - Fintech records
simulator/generators/fintech.py - Load scenarios
simulator/generators/scenarios.py
Lake ingestion
- Airflow orchestration
- Batch writer
simulator/generators/parquet_lakehouse.py - Raw Parquet lake
data_lake - Batch manifests
simulator/generators/parquet_lakehouse.py
Warehouse
- DuckDB raw views
duckdb/sql/bootstrap.sql - Audit tables
duckdb/sql/audit_tables.sql
Analytics models
- Staging models
dbt_fintech/models/staging - Intermediate models
dbt_fintech/models/intermediate - Core dimensions
dbt_fintech/models/marts/core - Payment facts
dbt_fintech/models/marts/core/fct_payment_events.sql - Settlement facts
dbt_fintech/models/marts/core/fct_settlements.sql - Chargeback facts
dbt_fintech/models/marts/core/fct_chargebacks.sql - Portfolio rollups
dbt_fintech/models/marts/core/mart_daily_portfolio.sql - Customer risk profile
dbt_fintech/models/intermediate/int_customer_risk_profile.sql - Payment lifecycle
dbt_fintech/models/intermediate/int_payment_lifecycle.sql
Other components
- Pipeline operator
Connections 17
- Pipeline operator → Airflow orchestration: triggers
README.md - Airflow orchestration → Synthetic API: requests batches
- Synthetic API → Fintech records: generates records
simulator/api_server.py - Airflow orchestration → Batch writer: orchestrates ingestion
- Batch writer → Fintech records: streams records
simulator/generators/parquet_lakehouse.py - Batch writer → Raw Parquet lake: writes partitions
README.md - Batch writer → Batch manifests: writes manifest
simulator/generators/parquet_lakehouse.py - Raw Parquet lake → DuckDB raw views: provides raw data
README.md - DuckDB raw views → Staging models: feeds staging
README.md - Staging models → Intermediate models: transforms
- Intermediate models → Core dimensions: curates
README.md - Core dimensions → Payment facts: includes
README.md - Core dimensions → Settlement facts: includes
README.md - Core dimensions → Chargeback facts: includes
README.md - Core dimensions → Portfolio rollups: includes
README.md - Intermediate models → Customer risk profile: includes
README.md - Intermediate models → Payment lifecycle: includes