# dangshark/market-risk-management-and-fraud-detection architecture

> Architecture diagram of the GitHub repository dangshark/market-risk-management-and-fraud-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dangshark/market-risk-management-and-fraud-detection
- Repository: https://github.com/dangshark/market-risk-management-and-fraud-detection

## Overview

1. **What this repo is**
   This is a containerized Big Data pipeline for two related domains: **market risk management** and **fraud detection**. The architecture is split into a **training lane** and a **real-time deployment/inference lane**, both coordinated by Airflow.

2. **Central orchestrator: Airflow**
   `airflow/dags/train.py` and `airflow/dags/deploy.py` define the two main workflows. Airflow is the control plane that starts/stops services, runs Spark jobs, and sequences Kafka-based data generation around model training and deployment. The logs show tasks such as `start_model_services`, `train_risk_models`, `train_fraud_model`, `start_fraud_producer`, and `start_fraud_detection_inference`.

3. **Training pipeline**
   The training DAG appears to:
   - stop any live model service before retraining,
   - start a fraud producer,
   - ingest or generate data through Kafka,
   - run Spark jobs for batch processing and feature preparation,
   - train the market-risk models and the fraud model,
   - persist the resulting artifacts.
   
   This lane is the batch side of the system and is likely the source of the persisted fraud model file in `model/fraud_xgb_pipeline.pkl`.

4. **Real-time deployment pipeline**
   The deploy DAG appears to bring up the runtime stack:
   - model services,
   - fraud producer,
   - fraud application/API,
   - fraud detection inference,
   - Kafka-based data flow for live scoring.
   
   This is the operational lane used after training finishes. The README explicitly says `training_pipeline` should succeed before `deploy_pipeline`.

5. **Kafka as the streaming backbone**
   `kafka-producer/producer.py` and the `fraud-inference` package show Kafka as the main event bus. The README describes daily price topics such as `daily_prices_GSPC`, `daily_prices_CVX`, `daily_prices_BP`, and `daily_prices_XOM`, and sample transaction-style data for fraud detection. Kafka is the boundary between data generation and downstream processing/scoring.

6. **Market risk model serving**
   `model_service/inference.py` is the runtime inference service for the market-risk side. The README says four models run in parallel for `GSPC`, `CVX`, `BP`, and `XOM`, which suggests company-specific model endpoints or model selection inside one service. The `model/` directory contains the persisted model artifact used by this service.

7. **Fraud detection stack**
   Fraud detection is implemented as a separate service chain:
   - `fraud-train/train_fraud_model.py` for training,
   - `fraud-inference/kafka_consumer.py` and `fraud-inference/main.py` for consuming streaming events and scoring them,
   - `fraud-inference/kafka_producer.py` suggests scores or downstream events may be published back to Kafka.
   
   The fraud path is therefore a streaming inference loop rather than just a one-shot model call.

8. **Spark and HDFS for batch data engineering**
   `spark-jobs/save_to_hdfs_job.py` and `spark-jobs/train_job.py` are the batch processing entry points. The README references Apache Spark/PySpark and HDFS, and the Airflow logs include `save_to_hdfs`, `train_risk_models`, and Spark UI checks. This indicates Spark handles both data persistence to HDFS and batch training/preprocessing.

9. **User-facing dashboard**
   `dashboard-api/main.py` plus `dashboard-api/static/index.html` is a lightweight UI/API layer for viewing system results. The README says results can be inspected in the UI at `localhost:8501` and also via service logs, so this component is probably a thin presentation layer over backend results rather than a heavy application server.

10. **Infrastructure and deployment boundaries**
   The whole system is containerized with `docker-compose.yml`, plus `Makefile` and `run.bat` for Linux/Windows lifecycle management. `HOST_DATA_PATH` in `.env` binds local Kaggle fraud data into containers. The repo also depends on external infrastructure referenced in the README: **Kafka**, **Spark**, **HDFS**, and **Redis** for low-latency result storage.

11. **How data flows end to end**
   The main flow is:
   - source data enters via Kafka producer or batch datasets,
   - Spark/HDFS handle storage and batch preparation,
   - training produces model artifacts,
   - Airflow deploys services,
   - streaming events go through Kafka to fraud inference and market-risk inference,
   - outputs are surfaced through logs and the dashboard/API.
   
   For an architecture diagram, the most important boundaries are **Airflow control plane**, **Kafka streaming bus**, **Spark/HDFS batch layer**, **model serving/inference services**, and the **dashboard/UI layer**.

Main entry points: [`airflow/dags/train.py`](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/train.py) (Train DAG), [`airflow/dags/deploy.py`](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/deploy.py) (Deploy DAG), [`fraud-inference/main.py`](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/main.py) (Fraud Inference), [`docker-compose.yml`](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/docker-compose.yml) (Compose Stack)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow_train["Train DAG<br/>Airflow DAG<br/>[train.py]"]
  node_airflow_deploy["Deploy DAG<br/>Airflow DAG<br/>[deploy.py]"]
end

subgraph group_batch["Batch Layer"]
  node_spark_save_hdfs["Save to HDFS<br/>Spark job"]
  node_spark_train_job["Train Job<br/>Spark job<br/>[train_job.py]"]
  node_fraud_train["Fraud Train<br/>ML trainer"]
end

subgraph group_streaming["Streaming Layer"]
  node_kafka_producer["Data Producer<br/>Kafka publisher<br/>[producer.py]"]
  node_fraud_consumer["Fraud Consumer<br/>stream consumer<br/>[kafka_consumer.py]"]
  node_fraud_main["Fraud Inference<br/>scoring service<br/>[main.py]"]
  node_fraud_pub["Fraud Publisher<br/>Kafka publisher<br/>[kafka_producer.py]"]
end

subgraph group_serving["Serving Layer"]
  node_risk_models[("Risk Models")]
  node_model_service["Risk Service<br/>inference service<br/>[inference.py]"]
end

subgraph group_presentation["Presentation"]
  node_dashboard_api["Dashboard API<br/>UI/API<br/>[main.py]"]
  node_dashboard_ui["Dashboard UI<br/>static frontend<br/>[index.html]"]
end

subgraph group_infra["Infrastructure"]
  node_docker_compose["Compose Stack<br/>container runtime<br/>[docker-compose.yml]"]
  node_makefile["Lifecycle Tools<br/>ops scripts"]
  node_run_bat["Windows Runner<br/>ops script<br/>[run.bat]"]
end

node_airflow_train -->|"generate data"| node_kafka_producer
node_airflow_train -->|"batch persist"| node_spark_save_hdfs
node_airflow_train -->|"prepare/train"| node_spark_train_job
node_airflow_train -->|"train fraud"| node_fraud_train
node_airflow_train -->|"stop/start"| node_model_service
node_spark_train_job -->|"produce artifacts"| node_risk_models
node_fraud_train -->|"store model"| node_risk_models
node_airflow_deploy -->|"start services"| node_model_service
node_airflow_deploy -->|"start producer"| node_kafka_producer
node_airflow_deploy -->|"start scoring"| node_fraud_main
node_airflow_deploy -->|"start consumer"| node_fraud_consumer
node_kafka_producer -->|"stream events"| node_fraud_consumer
node_fraud_consumer -->|"score event"| node_fraud_main
node_fraud_main -->|"emit result"| node_fraud_pub
node_fraud_main -->|"surface results"| node_dashboard_api
node_model_service -->|"surface outputs"| node_dashboard_api
node_dashboard_api -->|"serve ui"| node_dashboard_ui
node_docker_compose -->|"orchestrate"| node_airflow_train
node_docker_compose -->|"orchestrate"| node_airflow_deploy
node_makefile -->|"run stack"| node_docker_compose
node_run_bat -->|"run stack"| node_docker_compose

click node_airflow_train "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/train.py"
click node_airflow_deploy "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/deploy.py"
click node_kafka_producer "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/kafka-producer/producer.py"
click node_spark_save_hdfs "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/spark-jobs/save_to_hdfs_job.py"
click node_spark_train_job "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/spark-jobs/train_job.py"
click node_fraud_train "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-train/train_fraud_model.py"
click node_risk_models "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/model/fraud_xgb_pipeline.pkl"
click node_model_service "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/model_service/inference.py"
click node_fraud_consumer "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/kafka_consumer.py"
click node_fraud_main "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/main.py"
click node_fraud_pub "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/kafka_producer.py"
click node_dashboard_api "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/dashboard-api/main.py"
click node_dashboard_ui "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/dashboard-api/static/index.html"
click node_docker_compose "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/docker-compose.yml"
click node_makefile "https://github.com/dangshark/market-risk-management-and-fraud-detection/tree/main/Makefile"
click node_run_bat "https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/run.bat"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow_train,node_airflow_deploy toneBlue
class node_spark_save_hdfs,node_spark_train_job,node_fraud_train toneAmber
class node_kafka_producer,node_fraud_consumer,node_fraud_main,node_fraud_pub toneMint
class node_risk_models,node_model_service toneRose
class node_dashboard_api,node_dashboard_ui toneIndigo
class node_docker_compose,node_makefile,node_run_bat toneTeal
```

## Components

### Orchestration

Airflow control plane for training and deployment workflows.

- [Train DAG](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/train.py): Airflow DAG · Coordinates retraining, data generation, Spark jobs, and model training. · `airflow/dags/train.py`
- [Deploy DAG](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/airflow/dags/deploy.py): Airflow DAG · Brings up runtime services and streaming inference after training. · `airflow/dags/deploy.py`

### Batch Layer

Spark/HDFS training and data preparation path.

- [Save to HDFS](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/spark-jobs/save_to_hdfs_job.py): Spark job · Batch job that persists prepared data to HDFS. · `spark-jobs/save_to_hdfs_job.py`
- [Train Job](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/spark-jobs/train_job.py): Spark job · Batch preparation and training entry point for risk workflows. · `spark-jobs/train_job.py`
- [Fraud Train](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-train/train_fraud_model.py): ML trainer · Trains the fraud detection model and produces the persisted pipeline artifact. · `fraud-train/train_fraud_model.py`

### Streaming Layer

Kafka-based event generation and fraud scoring loop.

- [Data Producer](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/kafka-producer/producer.py): Kafka publisher · Generates or publishes market-risk and fraud events into Kafka. · `kafka-producer/producer.py`
- [Fraud Consumer](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/kafka_consumer.py): stream consumer · Consumes Kafka events for real-time fraud scoring. · `fraud-inference/kafka_consumer.py`
- [Fraud Inference](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/main.py): scoring service · Runs the fraud detection loop and application logic. · `fraud-inference/main.py`
- [Fraud Publisher](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/fraud-inference/kafka_producer.py): Kafka publisher · Publishes fraud-related downstream events or scores back to Kafka. · `fraud-inference/kafka_producer.py`

### Serving Layer

Runtime model services for market risk and fraud.

- [Risk Models](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/model/fraud_xgb_pipeline.pkl): market-risk models · Company-specific market-risk model set for the tracked symbols. · `model/fraud_xgb_pipeline.pkl`
- [Risk Service](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/model_service/inference.py): inference service · Serves market-risk inference for multiple assets in parallel. · `model_service/inference.py`

### Presentation

Dashboard and lightweight UI/API surface.

- [Dashboard API](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/dashboard-api/main.py): UI/API · Lightweight API and static frontend for viewing results. · `dashboard-api/main.py`
- [Dashboard UI](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/dashboard-api/static/index.html): static frontend · Browser interface for inspecting pipeline outputs and scores. · `dashboard-api/static/index.html`

### Infrastructure

Containerization and local lifecycle tooling.

- [Compose Stack](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/docker-compose.yml): container runtime · Defines the multi-service local deployment environment. · `docker-compose.yml`
- [Lifecycle Tools](https://github.com/dangshark/market-risk-management-and-fraud-detection/tree/main/Makefile): ops scripts · Convenience entry points for building and running the stack. · `Makefile`
- [Windows Runner](https://github.com/dangshark/market-risk-management-and-fraud-detection/blob/main/run.bat): ops script · Windows lifecycle helper for launching the environment. · `run.bat`

## Connections

- Train DAG → Data Producer: generate data · Training workflow starts the producer to feed batch and streaming inputs.
- Train DAG → Save to HDFS: batch persist · Training workflow triggers Spark persistence into HDFS.
- Train DAG → Train Job: prepare/train · Training workflow runs Spark-based batch preparation and training.
- Train DAG → Fraud Train: train fraud · Training workflow retrains the fraud model.
- Train DAG → Risk Service: stop/start · Training workflow manages market-risk serving lifecycle around retraining.
- Train Job → Risk Models: produce artifacts · Spark training path yields market-risk model artifacts.
- Fraud Train → Risk Models: store model · Fraud training persists the reusable fraud pipeline artifact.
- Deploy DAG → Risk Service: start services · Deployment workflow starts market-risk inference services.
- Deploy DAG → Data Producer: start producer · Deployment workflow launches the event producer.
- Deploy DAG → Fraud Inference: start scoring · Deployment workflow starts the fraud inference application.
- Deploy DAG → Fraud Consumer: start consumer · Deployment workflow starts Kafka consumption for fraud scoring.
- Data Producer → Fraud Consumer: stream events · Kafka producer feeds live events into the fraud consumer.
- Fraud Consumer → Fraud Inference: score event · Consumer hands incoming events to the fraud inference logic.
- Fraud Inference → Fraud Publisher: emit result · Fraud scoring service may publish results or alerts back to Kafka.
- Fraud Inference → Dashboard API: surface results · Fraud inference outputs are exposed through the dashboard/API layer.
- Risk Service → Dashboard API: surface outputs · Market-risk inference results are consumable by the dashboard/API.
- Dashboard API → Dashboard UI: serve ui · API backs the static dashboard interface.
- Compose Stack → Train DAG: orchestrate · Compose defines the training control-plane services.
- Compose Stack → Deploy DAG: orchestrate · Compose defines the deployment control-plane services.
- Lifecycle Tools → Compose Stack: run stack · Make targets manage the composed environment.
- Windows Runner → Compose Stack: run stack · Windows script launches the composed environment.
