How sanj123ay/fraud-lakehouse works
This is a local real-time fraud-detection lakehouse. Its main workflow generates transaction events, streams them through Kafka and Spark for validation, feature calculation, enrichment and scoring, then persists lakehouse data and operational predictions. Separate branches synchronize bank-database changes through Debezium and retrain the model nightly; Airflow also schedules Iceberg maintenance. The streaming source imports shared configuration and schemas. Some jobs and DAGs are represented by planning comments rather than implemented code, so those links reflect the README’s documented design, not verified runtime calls.
Main entry points: job1_streaming_features.py Transaction Streaming · create_topics.sh Kafka Topics · job2_train_model.py Batch Model Training · dag_01_daily_training.py Airflow Scheduler
Components 18
Ingestion and CDC
- Transaction Generator
nifi/scripts/transaction_generator.groovy - NiFi Flow
nifi/flow/README.md - Kafka Topics
kafka/scripts/create_topics.sh - Bank Database
docker/postgres-source/init/01_schema.sql - Debezium CDC
cdc/debezium/customers-connector.json - CDC Dimension Sync
spark_jobs/streaming/job3_cdc_customers_sync.py
Streaming and Features
- Transaction Streaming
spark_jobs/streaming/job1_streaming_features.py - Data Schemas
spark_jobs/common/schemas.py - Spark Configuration
spark_jobs/common/config.py
Lakehouse and Serving
- Iceberg Lakehouse
spark_jobs/streaming/job1_streaming_features.py - MinIO Storage
docker-compose.yml - Prediction PostgreSQL
spark_jobs/common/config.py - Latest Risk Scores
spark_jobs/common/config.py
Training and Operations
- Fraud Model Artifact
spark_jobs/common/config.py - Airflow Scheduler
airflow/dags/dag_01_daily_training.py - Batch Model Training
spark_jobs/batch/job2_train_model.py - Iceberg Maintenance
airflow/dags/dag_02_iceberg_maintenance.py
Other components
- Analysts
Connections 20
- Transaction Generator → NiFi Flow: generates events
README.md - NiFi Flow → Kafka Topics: publishes transactions
README.md - Kafka Topics → Transaction Streaming: streams transactions
README.md - Transaction Streaming → Data Schemas: parses with
spark_jobs/streaming/job1_streaming_features.py - Transaction Streaming → Spark Configuration: uses settings
spark_jobs/streaming/job1_streaming_features.py - Transaction Streaming → Iceberg Lakehouse: writes tables
README.md - Transaction Streaming → Iceberg Lakehouse: reads dimensions
spark_jobs/streaming/job1_streaming_features.py - Bank Database → Debezium CDC: streams changes
README.md - Debezium CDC → Kafka Topics: publishes CDC
README.md - Kafka Topics → CDC Dimension Sync: streams changes
README.md - CDC Dimension Sync → Iceberg Lakehouse: merges dimensions
README.md - Airflow Scheduler → Batch Model Training: runs nightly
README.md - Batch Model Training → Iceberg Lakehouse: reads history
README.md - Batch Model Training → Fraud Model Artifact: publishes model
README.md - Fraud Model Artifact → Transaction Streaming: loads for scoring
README.md - Transaction Streaming → Prediction PostgreSQL: writes predictions
README.md - Transaction Streaming → Latest Risk Scores: writes risk scores
README.md - Analysts → Prediction PostgreSQL: queries results
README.md - Airflow Scheduler → Iceberg Maintenance: schedules upkeep
README.md - Iceberg Lakehouse → MinIO Storage: stores table data
README.md