How dangshark/market-risk-management-and-fraud-detection works
1. What this repo is
This is a containerized Big Data pipeline for two related domains: market risk management and fraud detection. The architecture is split into a training lane and a real-time deployment/inference lane, both coordinated by Airflow.
2. Central orchestrator: Airflow
airflow/dags/train.py and airflow/dags/deploy.py define the two main workflows. Airflow is the control plane that starts/stops services, runs Spark jobs, and sequences Kafka-based data generation around model training and deployment. The logs show tasks such as start_model_services, train_risk_models, train_fraud_model, start_fraud_producer, and start_fraud_detection_inference.
3. Training pipeline
The training DAG appears to:
- stop any live model service before retraining,
- start a fraud producer,
- ingest or generate data through Kafka,
- run Spark jobs for batch processing and feature preparation,
- train the market-risk models and the fraud model,
- persist the resulting artifacts.
This lane is the batch side of the system and is likely the source of the persisted fraud model file in model/fraud_xgb_pipeline.pkl.
4. Real-time deployment pipeline
The deploy DAG appears to bring up the runtime stack:
- model services,
- fraud producer,
- fraud application/API,
- fraud detection inference,
- Kafka-based data flow for live scoring.
This is the operational lane used after training finishes. The README explicitly says training_pipeline should succeed before deploy_pipeline.
5. Kafka as the streaming backbone
kafka-producer/producer.py and the fraud-inference package show Kafka as the main event bus. The README describes daily price topics such as daily_prices_GSPC, daily_prices_CVX, daily_prices_BP, and daily_prices_XOM, and sample transaction-style data for fraud detection. Kafka is the boundary between data generation and downstream processing/scoring.
6. Market risk model serving
model_service/inference.py is the runtime inference service for the market-risk side. The README says four models run in parallel for GSPC, CVX, BP, and XOM, which suggests company-specific model endpoints or model selection inside one service. The model/ directory contains the persisted model artifact used by this service.
7. Fraud detection stack
Fraud detection is implemented as a separate service chain:
- fraud-train/train_fraud_model.py for training,
- fraud-inference/kafka_consumer.py and fraud-inference/main.py for consuming streaming events and scoring them,
- fraud-inference/kafka_producer.py suggests scores or downstream events may be published back to Kafka.
The fraud path is therefore a streaming inference loop rather than just a one-shot model call.
8. Spark and HDFS for batch data engineering
spark-jobs/save_to_hdfs_job.py and spark-jobs/train_job.py are the batch processing entry points. The README references Apache Spark/PySpark and HDFS, and the Airflow logs include save_to_hdfs, train_risk_models, and Spark UI checks. This indicates Spark handles both data persistence to HDFS and batch training/preprocessing.
9. User-facing dashboard
dashboard-api/main.py plus dashboard-api/static/index.html is a lightweight UI/API layer for viewing system results. The README says results can be inspected in the UI at localhost:8501 and also via service logs, so this component is probably a thin presentation layer over backend results rather than a heavy application server.
10. Infrastructure and deployment boundaries
The whole system is containerized with docker-compose.yml, plus Makefile and run.bat for Linux/Windows lifecycle management. HOST_DATA_PATH in .env binds local Kaggle fraud data into containers. The repo also depends on external infrastructure referenced in the README: Kafka, Spark, HDFS, and Redis for low-latency result storage.
11. How data flows end to end
The main flow is:
- source data enters via Kafka producer or batch datasets,
- Spark/HDFS handle storage and batch preparation,
- training produces model artifacts,
- Airflow deploys services,
- streaming events go through Kafka to fraud inference and market-risk inference,
- outputs are surfaced through logs and the dashboard/API.
For an architecture diagram, the most important boundaries are Airflow control plane, Kafka streaming bus, Spark/HDFS batch layer, model serving/inference services, and the dashboard/UI layer.
Main entry points: train.py Train DAG · deploy.py Deploy DAG · main.py Fraud Inference · docker-compose.yml Compose Stack
Components 16
Orchestration
Airflow control plane for training and deployment workflows.
- Train DAG · Coordinates retraining, data generation, Spark jobs, and model training.
airflow/dags/train.py - Deploy DAG · Brings up runtime services and streaming inference after training.
airflow/dags/deploy.py
Batch Layer
Spark/HDFS training and data preparation path.
- Save to HDFS · Batch job that persists prepared data to HDFS.
spark-jobs/save_to_hdfs_job.py - Train Job · Batch preparation and training entry point for risk workflows.
spark-jobs/train_job.py - Fraud Train · Trains the fraud detection model and produces the persisted pipeline artifact.
fraud-train/train_fraud_model.py
Streaming Layer
Kafka-based event generation and fraud scoring loop.
- Data Producer · Generates or publishes market-risk and fraud events into Kafka.
kafka-producer/producer.py - Fraud Consumer · Consumes Kafka events for real-time fraud scoring.
fraud-inference/kafka_consumer.py - Fraud Inference · Runs the fraud detection loop and application logic.
fraud-inference/main.py - Fraud Publisher · Publishes fraud-related downstream events or scores back to Kafka.
fraud-inference/kafka_producer.py
Serving Layer
Runtime model services for market risk and fraud.
- Risk Models · Company-specific market-risk model set for the tracked symbols.
model/fraud_xgb_pipeline.pkl - Risk Service · Serves market-risk inference for multiple assets in parallel.
model_service/inference.py
Presentation
Dashboard and lightweight UI/API surface.
- Dashboard API · Lightweight API and static frontend for viewing results.
dashboard-api/main.py - Dashboard UI · Browser interface for inspecting pipeline outputs and scores.
dashboard-api/static/index.html
Infrastructure
Containerization and local lifecycle tooling.
- Compose Stack · Defines the multi-service local deployment environment.
docker-compose.yml - Lifecycle Tools · Convenience entry points for building and running the stack.
Makefile - Windows Runner · Windows lifecycle helper for launching the environment.
run.bat
Connections 21
- Train DAG → Data Producer: generate data · Training workflow starts the producer to feed batch and streaming inputs.
- Train DAG → Save to HDFS: batch persist · Training workflow triggers Spark persistence into HDFS.
- Train DAG → Train Job: prepare/train · Training workflow runs Spark-based batch preparation and training.
- Train DAG → Fraud Train: train fraud · Training workflow retrains the fraud model.
- Train DAG → Risk Service: stop/start · Training workflow manages market-risk serving lifecycle around retraining.
- Train Job → Risk Models: produce artifacts · Spark training path yields market-risk model artifacts.
- Fraud Train → Risk Models: store model · Fraud training persists the reusable fraud pipeline artifact.
- Deploy DAG → Risk Service: start services · Deployment workflow starts market-risk inference services.
- Deploy DAG → Data Producer: start producer · Deployment workflow launches the event producer.
- Deploy DAG → Fraud Inference: start scoring · Deployment workflow starts the fraud inference application.
- Deploy DAG → Fraud Consumer: start consumer · Deployment workflow starts Kafka consumption for fraud scoring.
- Data Producer → Fraud Consumer: stream events · Kafka producer feeds live events into the fraud consumer.
- Fraud Consumer → Fraud Inference: score event · Consumer hands incoming events to the fraud inference logic.
- Fraud Inference → Fraud Publisher: emit result · Fraud scoring service may publish results or alerts back to Kafka.
- Fraud Inference → Dashboard API: surface results · Fraud inference outputs are exposed through the dashboard/API layer.
- Risk Service → Dashboard API: surface outputs · Market-risk inference results are consumable by the dashboard/API.
- Dashboard API → Dashboard UI: serve ui · API backs the static dashboard interface.
- Compose Stack → Train DAG: orchestrate · Compose defines the training control-plane services.
- Compose Stack → Deploy DAG: orchestrate · Compose defines the deployment control-plane services.
- Lifecycle Tools → Compose Stack: run stack · Make targets manage the composed environment.
- Windows Runner → Compose Stack: run stack · Windows script launches the composed environment.