How suvkp/fraud_detection works
Main Components
• Data Collection & Ingestion:
• Represented by the modules in src: data_loader.py for loading input data.
• Data Preprocessing:
• Handled by src/data_preprocessing.py, which cleans and prepares the data.
• Exploratory Data Analysis (EDA):
• Notebooks in the notebook directory (01_eda.ipynb) and src/eda.py provide analysis and insights.
• Feature Engineering / Feature Selection:
• The src/feature_selection.py module highlights data/features chosen for model training.
• Model Training:
• Managed through src/model_training.py and pipeline/train_pipeline.py. This part includes training models using the engineered features.
• Model Evaluation:
• Implemented in src/model_evaluation.py to assess model performance and benchmark different approaches.
• Model Inference (Deployment):
• Handled by src/model_inference.py and pipeline/inference_pipeline.py for running predictions on unseen data.
• Utility Functions:
• Supporting functions in src/util.py that may be used across the various modules.
• Entry Point:
• The run_pipeline.py script serves as the primary trigger for executing training or inference pipelines.
Relationships and Interactions
• Data flows from loading (data_loader.py) to preprocessing (data_preprocessing.py).
• Preprocessed data is then explored (EDA notebooks and src/eda.py) and refined via feature selection.
• The refined dataset is split into two pipelines:
• Training pipeline: where the model is trained using src/model_training.py and pipeline/train_pipeline.py.
• Inference pipeline: where the trained model is used for prediction via src/model_inference.py and pipeline/inference_pipeline.py.
• Model evaluation (in src/model_evaluation.py) acts as a checkpoint that feeds back into training for potential improvements.
• Utility functions (in src/util.py) support all these stages.
Architectural Patterns and Design Principles
• Modular separation of concerns: Each stage of the machine learning workflow is encapsulated in its own file/module.
• Pipeline-based design: Separate directories for training (train_pipeline) and inference (inference_pipeline) enforce clarity in workflow execution.
• Use of notebooks for early-stage EDA ensures that exploratory analysis is not mixed with production pipelines.
• The design is linear in terms of data flow yet modular enough to allow for independent unit testing and iterative improvements.
Key Technologies and Tools
• Python (as evident from .py files) is the core language.
• Jupyter Notebooks for EDA and early-stage development.
• Machine Learning frameworks and libraries (implied by the context) for model training, evaluation, and inference.
• Version control (given the .gitignore), indicating a mature development process.