How chuksoo/sneaky-swipe-detector works
Project Type and Purpose
• This is a machine‐learning‐driven fraud detection service built in Python. Its main goal is to ingest transaction data, preprocess it, train a classification model to spot fraudulent “sneaky swipes,” and then make predictions on new data.
• Key features:
• Data ingestion & storage (raw Parquet files)
• Data preprocessing & feature engineering
• Model training pipeline
• Model inference/prediction pipeline
• Supporting notebooks for exploration and reports for documentation
File Structure Analysis
• Top‐level directories:
• data/raw: raw transaction datasets (Parquet)
• environment: setup docs and Conda/YAML files
• notebooks: Jupyter notebooks for exploratory data analysis
• reports: PDF/TeX end-to-end project plan
• scripts: operational scripts (data_download.py, train.py, predict.py, plotting.py)
• src/fraud_detection: core Python package (main service logic + utility modules)
• src/tests: unit tests for models and utilities
• Patterns & conventions:
• Scripts orchestrate each stage (download → preprocess → train → predict)
• Core logic isolated under src/fraud_detection (separation of concerns: data, utils, main entry point)
• Tests mirror code structure under src/tests for CI coverage
Key Insights from README
• Emphasis on a modular pipeline (data → features → modeling → API if added in future)
• Plans for orchestration (Prefect/Airflow) and deployment (Docker/Kubernetes), though not yet present
• Clear separation of concerns: development (notebooks), operations (scripts), packaging (setup.py), testing
a. Main Components
1. Data Store
– Raw Data (Parquet files under data/raw)
– Artifacts (trained model files saved locally or in a models/ directory)
2. Data Ingestion & Preprocessing
– script: data_download.py
– module: fraud_detection.utils.data_preprocessing
3. Feature Engineering & Utility Functions
– module: fraud_detection.utils.utils
4. Model Training Pipeline
– script: train.py
– entry point: src/fraud_detection/main.py (train flow)
5. Model Serving / Inference
– script: predict.py
– entry point: src/fraud_detection/main.py (predict flow)
6. Notebooks & Reports
– notebooks: exploratory analysis (sneaky-swipe-detect.ipynb)
– reports: project plan documentation
7. Testing & CI
– src/tests/test_models.py, src/tests/utils.py
8. Environment & Packaging
– environment/*.yml, setup.py, requirements.txt
b. Relationships and Data Flow
1. Data Download → Raw Data Store
2. Raw Data → Preprocessing Module → Processed Data
3. Processed Data → Feature Engineering → Training Routine
4. Training Routine → Trained Model Artifact
5. Trained Model + New Input Data → Inference Routine → Predictions
6. Notebooks read Raw/Processed Data & Trained Model for analysis and visualization
7. Tests validate utility modules and model outputs
c. Architectural Patterns & Design Principles
• Modular pipeline: each stage separated into scripts and library code
• Single-responsibility: preprocessing, training, inference isolated
• Test-driven development: mirrors code structure in src/tests
• Future-ready for orchestration/deployment: clear boundaries facilitate adding Prefect/Airflow or Docker
d. Technologies & Frameworks
• Python 3.x
• pandas, scikit-learn (or similar) for data processing & modeling
• parquet for data storage
• pytest (or unittest) for testing
• Conda/YAML for environment reproducibility