How bhagyesh-074/fraudstream works
FraudStream detects credit-card fraud through offline model development and Kafka-based real-time scoring. The primary runtime path replays transactions into Kafka, scores each event with shared feature logic and an XGBoost model, publishes scored events, and exposes telemetry. Separate branches ingest raw events into a Parquet lake and monitor feature drift; Airflow can trigger retraining and registry promotion gating. The README documents several integrations and flows; source excerpts are partial, so uncertain internal links are omitted.
Main entry points: train.py Model training · run_eda.py EDA runner · scoring_consumer.py Scoring consumer · retrain_dag.py Retraining DAG
Components 25
Data exploration
- Dataset check
src/eda/download_data.py - Data profiling
src/eda/profiling.py - Class imbalance
src/eda/imbalance.py - Fraud cost model
src/eda/cost_matrix.py - EDA runner
src/eda/run_eda.py
Model development
- Data cleaning
src/pipeline/cleaning.py - Shared features
src/features/feature_logic.py - Training features
src/pipeline/features.py - Feature store
src/features/feature_store.py - Model evaluation
src/pipeline/evaluate.py - Model training
src/pipeline/train.py - Champion model
models/xgboost_baseline.json.dvc
Event processing
- Kafka producer
src/streaming/producer.py - Kafka broker
- Raw event consumer
src/streaming/raw_consumer.py - Partitioned Parquet lake
data
Scoring and telemetry
- Model loader
src/serving/model_loader.py - Scoring consumer
src/serving/scoring_consumer.py - Scored events
- Telemetry API
src/serving/api.py
Monitoring and retraining
- Drift detector
src/monitoring/drift_detector.py - Retraining DAG
src/orchestration/retrain_dag.py - MLflow registry
Other components
- Transaction dataset
- Operator
Connections 35
- Transaction dataset → Dataset check: checks availability
- EDA runner → Dataset check: checks dataset
src/eda/run_eda.py - EDA runner → Data profiling: profiles data
src/eda/run_eda.py - EDA runner → Class imbalance: reports imbalance
src/eda/run_eda.py - EDA runner → Fraud cost model: calibrates costs
src/eda/run_eda.py - Operator → EDA runner: runs analysis
README.md - Operator → Model training: trains model
README.md - Model training → Data cleaning: uses cleaning
src/pipeline/train.py - Model training → Training features: uses features
src/pipeline/train.py - Model training → Model evaluation: evaluates models
src/pipeline/train.py - Training features → Data cleaning: prepares features
src/pipeline/features.py - Data cleaning → Shared features: computes features
src/pipeline/cleaning.py - Model training → Data profiling: uses profiling
src/pipeline/train.py - Model training → Fraud cost model: uses cost model
src/pipeline/train.py - Model training → Feature store: writes feature data
README.md - Model training → Champion model: saves champion
README.md - Operator → Kafka producer: replays transactions
README.md - Kafka producer → Kafka broker: publishes raw events
README.md - Kafka broker → Raw event consumer: delivers raw events
README.md - Raw event consumer → Partitioned Parquet lake: writes partitions
README.md - Kafka broker → Scoring consumer: delivers transactions
README.md - Scoring consumer → Shared features: computes features
src/serving/scoring_consumer.py - Scoring consumer → Model loader: loads model
- Scoring consumer → Scored events: publishes scores
README.md - Telemetry API → Scoring consumer: serves scoring stats
src/serving/api.py - Telemetry API → Model loader: reports model status
src/serving/api.py - Operator → Telemetry API: queries telemetry
README.md - Feature store → Drift detector: provides baseline
README.md - Operator → Drift detector: runs drift check
README.md - Drift detector → Retraining DAG: triggers retraining
README.md - Retraining DAG → Drift detector: runs drift check
src/orchestration/retrain_dag.py - Retraining DAG → Model training: trains candidate
src/orchestration/retrain_dag.py - Retraining DAG → MLflow registry: gates promotion
README.md - Model training → MLflow registry: registers candidate
README.md - MLflow registry → Model loader: supplies production model