# sanj123ay/fraud-lakehouse architecture

> Architecture diagram of the GitHub repository sanj123ay/fraud-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanj123ay/fraud-lakehouse
- Repository: https://github.com/sanj123ay/fraud-lakehouse

## Overview

This is a local real-time fraud-detection lakehouse. Its main workflow generates transaction events, streams them through Kafka and Spark for validation, feature calculation, enrichment and scoring, then persists lakehouse data and operational predictions. Separate branches synchronize bank-database changes through Debezium and retrain the model nightly; Airflow also schedules Iceberg maintenance. The streaming source imports shared configuration and schemas. Some jobs and DAGs are represented by planning comments rather than implemented code, so those links reflect the README’s documented design, not verified runtime calls.

Main entry points: [`spark_jobs/streaming/job1_streaming_features.py`](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job1_streaming_features.py) (Transaction Streaming), [`kafka/scripts/create_topics.sh`](https://github.com/sanj123ay/fraud-lakehouse/blob/main/kafka/scripts/create_topics.sh) (Kafka Topics), [`spark_jobs/batch/job2_train_model.py`](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/batch/job2_train_model.py) (Batch Model Training), [`airflow/dags/dag_01_daily_training.py`](https://github.com/sanj123ay/fraud-lakehouse/blob/main/airflow/dags/dag_01_daily_training.py) (Airflow Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and CDC"]
  node_generator["Transaction Generator"]
  node_nifi["NiFi Flow<br/>[README.md]"]
  node_kafka["Kafka Topics<br/>[create_topics.sh]"]
  node_bankdb[("Bank Database<br/>[01_schema.sql]")]
  node_debezium["Debezium CDC"]
  node_cdc_sync["CDC Dimension Sync"]
end

subgraph group_stream["Streaming and Features"]
  node_stream_job["Transaction Streaming"]
  node_schemas["Data Schemas<br/>[schemas.py]"]
  node_config["Spark Configuration<br/>[config.py]"]
end

subgraph group_lakehouse["Lakehouse and Serving"]
  node_lakehouse[("Iceberg Lakehouse")]
  node_minio[("MinIO Storage<br/>[docker-compose.yml]")]
  node_predictions[("Prediction PostgreSQL<br/>[config.py]")]
  node_redis[("Latest Risk Scores<br/>[config.py]")]
end

subgraph group_mlops["Training and Operations"]
  node_model["Fraud Model Artifact<br/>[config.py]"]
  node_airflow["Airflow Scheduler"]
  node_training["Batch Model Training"]
  node_maintenance["Iceberg Maintenance"]
end

node_analysts(("Analysts"))

node_generator -->|"generates events"| node_nifi
node_nifi -->|"publishes transactions"| node_kafka
node_kafka -->|"streams transactions"| node_stream_job
node_stream_job -->|"parses with"| node_schemas
node_stream_job -->|"uses settings"| node_config
node_stream_job -->|"writes tables"| node_lakehouse
node_stream_job -->|"reads dimensions"| node_lakehouse
node_bankdb -->|"streams changes"| node_debezium
node_debezium -->|"publishes CDC"| node_kafka
node_kafka -->|"streams changes"| node_cdc_sync
node_cdc_sync -->|"merges dimensions"| node_lakehouse
node_airflow -->|"runs nightly"| node_training
node_training -->|"reads history"| node_lakehouse
node_training -->|"publishes model"| node_model
node_model -->|"loads for scoring"| node_stream_job
node_stream_job -->|"writes predictions"| node_predictions
node_stream_job -->|"writes risk scores"| node_redis
node_analysts -->|"queries results"| node_predictions
node_airflow -->|"schedules upkeep"| node_maintenance
node_lakehouse -->|"stores table data"| node_minio

click node_generator "https://github.com/sanj123ay/fraud-lakehouse/blob/main/nifi/scripts/transaction_generator.groovy"
click node_nifi "https://github.com/sanj123ay/fraud-lakehouse/blob/main/nifi/flow/README.md"
click node_kafka "https://github.com/sanj123ay/fraud-lakehouse/blob/main/kafka/scripts/create_topics.sh"
click node_bankdb "https://github.com/sanj123ay/fraud-lakehouse/blob/main/docker/postgres-source/init/01_schema.sql"
click node_debezium "https://github.com/sanj123ay/fraud-lakehouse/blob/main/cdc/debezium/customers-connector.json"
click node_cdc_sync "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job3_cdc_customers_sync.py"
click node_stream_job "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job1_streaming_features.py"
click node_schemas "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/schemas.py"
click node_config "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py"
click node_lakehouse "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job1_streaming_features.py"
click node_minio "https://github.com/sanj123ay/fraud-lakehouse/blob/main/docker-compose.yml"
click node_model "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py"
click node_airflow "https://github.com/sanj123ay/fraud-lakehouse/blob/main/airflow/dags/dag_01_daily_training.py"
click node_training "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/batch/job2_train_model.py"
click node_maintenance "https://github.com/sanj123ay/fraud-lakehouse/blob/main/airflow/dags/dag_02_iceberg_maintenance.py"
click node_predictions "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py"
click node_redis "https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_nifi,node_kafka,node_bankdb,node_debezium,node_cdc_sync toneBlue
class node_stream_job,node_schemas,node_config toneAmber
class node_lakehouse,node_minio,node_predictions,node_redis toneMint
class node_model,node_airflow,node_training,node_maintenance toneRose
class node_analysts toneIndigo
```

## Components

### Ingestion and CDC

- [Transaction Generator](https://github.com/sanj123ay/fraud-lakehouse/blob/main/nifi/scripts/transaction_generator.groovy): `nifi/scripts/transaction_generator.groovy`
- [NiFi Flow](https://github.com/sanj123ay/fraud-lakehouse/blob/main/nifi/flow/README.md): `nifi/flow/README.md`
- [Kafka Topics](https://github.com/sanj123ay/fraud-lakehouse/blob/main/kafka/scripts/create_topics.sh): `kafka/scripts/create_topics.sh`
- [Bank Database](https://github.com/sanj123ay/fraud-lakehouse/blob/main/docker/postgres-source/init/01_schema.sql): `docker/postgres-source/init/01_schema.sql`
- [Debezium CDC](https://github.com/sanj123ay/fraud-lakehouse/blob/main/cdc/debezium/customers-connector.json): `cdc/debezium/customers-connector.json`
- [CDC Dimension Sync](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job3_cdc_customers_sync.py): `spark_jobs/streaming/job3_cdc_customers_sync.py`

### Streaming and Features

- [Transaction Streaming](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job1_streaming_features.py): `spark_jobs/streaming/job1_streaming_features.py`
- [Data Schemas](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/schemas.py): `spark_jobs/common/schemas.py`
- [Spark Configuration](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py): `spark_jobs/common/config.py`

### Lakehouse and Serving

- [Iceberg Lakehouse](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/streaming/job1_streaming_features.py): `spark_jobs/streaming/job1_streaming_features.py`
- [MinIO Storage](https://github.com/sanj123ay/fraud-lakehouse/blob/main/docker-compose.yml): `docker-compose.yml`
- [Prediction PostgreSQL](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py): `spark_jobs/common/config.py`
- [Latest Risk Scores](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py): `spark_jobs/common/config.py`

### Training and Operations

- [Fraud Model Artifact](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/common/config.py): `spark_jobs/common/config.py`
- [Airflow Scheduler](https://github.com/sanj123ay/fraud-lakehouse/blob/main/airflow/dags/dag_01_daily_training.py): `airflow/dags/dag_01_daily_training.py`
- [Batch Model Training](https://github.com/sanj123ay/fraud-lakehouse/blob/main/spark_jobs/batch/job2_train_model.py): `spark_jobs/batch/job2_train_model.py`
- [Iceberg Maintenance](https://github.com/sanj123ay/fraud-lakehouse/blob/main/airflow/dags/dag_02_iceberg_maintenance.py): `airflow/dags/dag_02_iceberg_maintenance.py`

### Other components

- **Analysts**

## Connections

- Transaction Generator → NiFi Flow: generates events (evidence: `README.md`)
- NiFi Flow → Kafka Topics: publishes transactions (evidence: `README.md`)
- Kafka Topics → Transaction Streaming: streams transactions (evidence: `README.md`)
- Transaction Streaming → Data Schemas: parses with (evidence: `spark_jobs/streaming/job1_streaming_features.py`)
- Transaction Streaming → Spark Configuration: uses settings (evidence: `spark_jobs/streaming/job1_streaming_features.py`)
- Transaction Streaming → Iceberg Lakehouse: writes tables (evidence: `README.md`)
- Transaction Streaming → Iceberg Lakehouse: reads dimensions (evidence: `spark_jobs/streaming/job1_streaming_features.py`)
- Bank Database → Debezium CDC: streams changes (evidence: `README.md`)
- Debezium CDC → Kafka Topics: publishes CDC (evidence: `README.md`)
- Kafka Topics → CDC Dimension Sync: streams changes (evidence: `README.md`)
- CDC Dimension Sync → Iceberg Lakehouse: merges dimensions (evidence: `README.md`)
- Airflow Scheduler → Batch Model Training: runs nightly (evidence: `README.md`)
- Batch Model Training → Iceberg Lakehouse: reads history (evidence: `README.md`)
- Batch Model Training → Fraud Model Artifact: publishes model (evidence: `README.md`)
- Fraud Model Artifact → Transaction Streaming: loads for scoring (evidence: `README.md`)
- Transaction Streaming → Prediction PostgreSQL: writes predictions (evidence: `README.md`)
- Transaction Streaming → Latest Risk Scores: writes risk scores (evidence: `README.md`)
- Analysts → Prediction PostgreSQL: queries results (evidence: `README.md`)
- Airflow Scheduler → Iceberg Maintenance: schedules upkeep (evidence: `README.md`)
- Iceberg Lakehouse → MinIO Storage: stores table data (evidence: `README.md`)
