# omarfarooq908/fintech-lakehouse-pipeline architecture

> Architecture diagram of the GitHub repository omarfarooq908/fintech-lakehouse-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/omarfarooq908/fintech-lakehouse-pipeline
- Repository: https://github.com/omarfarooq908/fintech-lakehouse-pipeline

## Overview

This is a synthetic fintech data lakehouse: configurable fintech records feed partitioned Parquet, DuckDB raw views, and dbt models for curated analytics and risk. The principal workflow is batch generation and ingestion followed by staging, intermediate transformations, and marts; Airflow is documented as the orchestrator, but its source is absent from the tree. The graph preserves documented outputs and reliability subsystems, while limiting internal transformation links where source excerpts do not show model dependencies.

Main entry points: [`dbt_fintech/models/marts/core`](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/marts/core) (Core dimensions), [`simulator/generators/parquet_lakehouse.py`](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/parquet_lakehouse.py) (Batch writer), [`dbt_fintech/models/intermediate`](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/intermediate) (Intermediate models), [`simulator/api_server.py`](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/api_server.py) (Synthetic API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Synthetic data"]
  node_simulator_api["Synthetic API<br/>[api_server.py]"]
  node_record_generation["Fintech records<br/>[fintech.py]"]
  node_scenarios["Load scenarios<br/>[scenarios.py]"]
end

subgraph group_ingestion["Lake ingestion"]
  node_airflow["Airflow orchestration"]
  node_parquet_writer["Batch writer"]
  node_raw_lake[("Raw Parquet lake")]
  node_manifests["Batch manifests"]
end

subgraph group_warehouse["Warehouse"]
  node_duckdb_raw[("DuckDB raw views<br/>[bootstrap.sql]")]
  node_duckdb_audit[("Audit tables<br/>[audit_tables.sql]")]
end

subgraph group_analytics["Analytics models"]
  node_dbt_staging["Staging models"]
  node_dbt_intermediate["Intermediate models"]
  node_dbt_core["Core dimensions"]
  node_payment_facts["Payment facts"]
  node_settlement_facts["Settlement facts"]
  node_chargeback_facts["Chargeback facts"]
  node_portfolio_mart["Portfolio rollups"]
  node_risk_profile["Customer risk profile"]
  node_payment_lifecycle["Payment lifecycle"]
end

node_operator(("Pipeline operator"))

node_operator -->|"triggers"| node_airflow
node_airflow -.->|"requests batches"| node_simulator_api
node_simulator_api -->|"generates records"| node_record_generation
node_airflow -.->|"orchestrates ingestion"| node_parquet_writer
node_parquet_writer -->|"streams records"| node_record_generation
node_parquet_writer -->|"writes partitions"| node_raw_lake
node_parquet_writer -->|"writes manifest"| node_manifests
node_raw_lake -->|"provides raw data"| node_duckdb_raw
node_duckdb_raw -->|"feeds staging"| node_dbt_staging
node_dbt_staging -.->|"transforms"| node_dbt_intermediate
node_dbt_intermediate -->|"curates"| node_dbt_core
node_dbt_core -->|"includes"| node_payment_facts
node_dbt_core -->|"includes"| node_settlement_facts
node_dbt_core -->|"includes"| node_chargeback_facts
node_dbt_core -->|"includes"| node_portfolio_mart
node_dbt_intermediate -->|"includes"| node_risk_profile
node_dbt_intermediate -.->|"includes"| node_payment_lifecycle

click node_simulator_api "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/api_server.py"
click node_record_generation "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/fintech.py"
click node_scenarios "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/scenarios.py"
click node_parquet_writer "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/parquet_lakehouse.py"
click node_raw_lake "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/data_lake"
click node_manifests "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/parquet_lakehouse.py"
click node_duckdb_raw "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/duckdb/sql/bootstrap.sql"
click node_duckdb_audit "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/duckdb/sql/audit_tables.sql"
click node_dbt_staging "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/staging"
click node_dbt_intermediate "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/intermediate"
click node_dbt_core "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/marts/core"
click node_payment_facts "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_payment_events.sql"
click node_settlement_facts "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_settlements.sql"
click node_chargeback_facts "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_chargebacks.sql"
click node_portfolio_mart "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/mart_daily_portfolio.sql"
click node_risk_profile "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/intermediate/int_customer_risk_profile.sql"
click node_payment_lifecycle "https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/intermediate/int_payment_lifecycle.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simulator_api,node_record_generation,node_scenarios toneBlue
class node_airflow,node_parquet_writer,node_raw_lake,node_manifests toneAmber
class node_duckdb_raw,node_duckdb_audit toneMint
class node_dbt_staging,node_dbt_intermediate,node_dbt_core,node_payment_facts,node_settlement_facts,node_chargeback_facts,node_portfolio_mart,node_risk_profile,node_payment_lifecycle toneRose
class node_operator toneIndigo
```

## Components

### Synthetic data

- [Synthetic API](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/api_server.py): `simulator/api_server.py`
- [Fintech records](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/fintech.py): `simulator/generators/fintech.py`
- [Load scenarios](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/scenarios.py): `simulator/generators/scenarios.py`

### Lake ingestion

- **Airflow orchestration**
- [Batch writer](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/parquet_lakehouse.py): `simulator/generators/parquet_lakehouse.py`
- [Raw Parquet lake](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/data_lake): `data_lake`
- [Batch manifests](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/simulator/generators/parquet_lakehouse.py): `simulator/generators/parquet_lakehouse.py`

### Warehouse

- [DuckDB raw views](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/duckdb/sql/bootstrap.sql): `duckdb/sql/bootstrap.sql`
- [Audit tables](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/duckdb/sql/audit_tables.sql): `duckdb/sql/audit_tables.sql`

### Analytics models

- [Staging models](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/staging): `dbt_fintech/models/staging`
- [Intermediate models](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/intermediate): `dbt_fintech/models/intermediate`
- [Core dimensions](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/tree/main/dbt_fintech/models/marts/core): `dbt_fintech/models/marts/core`
- [Payment facts](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_payment_events.sql): `dbt_fintech/models/marts/core/fct_payment_events.sql`
- [Settlement facts](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_settlements.sql): `dbt_fintech/models/marts/core/fct_settlements.sql`
- [Chargeback facts](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/fct_chargebacks.sql): `dbt_fintech/models/marts/core/fct_chargebacks.sql`
- [Portfolio rollups](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/marts/core/mart_daily_portfolio.sql): `dbt_fintech/models/marts/core/mart_daily_portfolio.sql`
- [Customer risk profile](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/intermediate/int_customer_risk_profile.sql): `dbt_fintech/models/intermediate/int_customer_risk_profile.sql`
- [Payment lifecycle](https://github.com/omarfarooq908/fintech-lakehouse-pipeline/blob/main/dbt_fintech/models/intermediate/int_payment_lifecycle.sql): `dbt_fintech/models/intermediate/int_payment_lifecycle.sql`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Airflow orchestration: triggers (evidence: `README.md`)
- Airflow orchestration → Synthetic API: requests batches
- Synthetic API → Fintech records: generates records (evidence: `simulator/api_server.py`)
- Airflow orchestration → Batch writer: orchestrates ingestion
- Batch writer → Fintech records: streams records (evidence: `simulator/generators/parquet_lakehouse.py`)
- Batch writer → Raw Parquet lake: writes partitions (evidence: `README.md`)
- Batch writer → Batch manifests: writes manifest (evidence: `simulator/generators/parquet_lakehouse.py`)
- Raw Parquet lake → DuckDB raw views: provides raw data (evidence: `README.md`)
- DuckDB raw views → Staging models: feeds staging (evidence: `README.md`)
- Staging models → Intermediate models: transforms
- Intermediate models → Core dimensions: curates (evidence: `README.md`)
- Core dimensions → Payment facts: includes (evidence: `README.md`)
- Core dimensions → Settlement facts: includes (evidence: `README.md`)
- Core dimensions → Chargeback facts: includes (evidence: `README.md`)
- Core dimensions → Portfolio rollups: includes (evidence: `README.md`)
- Intermediate models → Customer risk profile: includes (evidence: `README.md`)
- Intermediate models → Payment lifecycle: includes