How arpitshourya/creditcardfrauddetection works
Project type and purpose
• This is an end-to-end credit‐card fraud detection system combining both streaming and batch ML pipelines plus an online inference API.
• Key features:
• Data ingestion from external sources into Kafka
• Real‐time scoring via Faust stream processor
• Batch training and batch prediction pipelines
• Model artifact storage and sync with AWS S3
• Online inference exposed via FastAPI
• Persistence of results (e.g., MongoDB)
File structure
• Top‐level service & orchestration:
• docker-compose.yml
• Dockerfile-fastapi (for API service)
• Dockerfile-faust (for stream processor)
• Streaming components (FraudDetection/components):
• kafka_producer.py / extract_push_data.py → produces transactions to Kafka
• faust_app.py / kafka_consumer.py → stream processing and real‐time inference
• kafka_producer / consumer wrappers
• Batch pipelines (FraudDetection/pipeline):
• training_pipeline.py → data ingestion → transformation → model training → artifact creation
• batch_prediction_pipeline.py → batch scoring on new data
• Model artifacts & schema:
• final_models/*.pkl
• data_schema/schema.yaml
• cloud/s3_sync.py → sync artifacts to/from AWS S3
• API layer:
• app.py / main.py → FastAPI server that loads model & preprocessing objects and serves /predict endpoints
• ping_mongo.py → health check for MongoDB
• Utilities:
• FraudDetection/utils (generic helpers, ML metrics, estimators)
• Cross‐cutting concerns:
• FraudDetection/logging/logger.py
• FraudDetection/exception