How allmeidaapedro/lending-club-credit-scoring works
Project type and purpose
• This is an open-source machine-learning pipeline for credit risk modeling (“Lending Club Credit Scoring”).
• It implements the CRISP-DM process: business understanding ➔ data understanding & cleaning ➔ feature engineering ➔ modeling (PD, EAD, LGD) ➔ validation & scorecard generation ➔ model monitoring ➔ final policy.
• Artifacts (trained models, scorecards), raw inputs (parquet, Excel), Jupyter notebooks, utility code (src/), and visualization reports compose the project.
File structure
• Top level:
• artifacts/ – trained models & scorecards (PKL, CSV)
• input/ – raw & cleaned data (Excel, parquet)
• notebooks/ – exploratory, modeling, monitoring notebooks
• reports/ – static images (plots, logos)
• src/ – reusable Python modules (eda_utils, modelling_utils, logger, exception)
• setup.py, requirements.txt – packaging & dependencies
• Pattern: sequential pipeline driven by notebooks, with shared utility code in src/.
README for additional insights
• CRISP-DM six-step workflow
• Technologies: Python, pandas, NumPy, scikit-learn, statsmodels, Jupyter
• Outputs: PD scorecard, EAD & LGD models, expected loss calculator, credit policy rules, monitoring via PSI.
Main components
a. Data Sources
• External raw data on Kaggle (downloaded manually)
• input/LCDataDictionary.xlsx, input/clean_loan_data.parquet
b. Data Ingestion & Cleaning
• notebooks/1_data_cleaning_understanding.ipynb
• src/eda_utils.py, src/logger.py, src/exception.py
• Outputs cleaned_parquet → input/clean_loan_data.parquet
c. Exploratory Data Analysis
• notebooks/2_eda.ipynb
• src/eda_utils.py
• Generates plots → reports/
d. Feature Engineering & Preprocessing
• src/modelling_utils.py
• Dummy creation, discretization, encoding, scaling
e. Model Training
• PD Modeling (notebooks/3_pd_modeling.ipynb) → logistic regression + scorecard → artifacts/pd_model/scorecard.csv
• EAD & LGD Modeling (notebooks/4_lgd_ead_modeling.ipynb) → two-stage LGD, linear EAD → artifacts/ead_lgd_models/*.pkl
f. Expected Loss & Credit Policy Engine
• Notebook combines PD, EAD, LGD outputs → calculates EL = PD·EAD·LGD
• Risk classes & policy rules → reports/
g. Model Monitoring
• notebooks/5_pd_model_monitoring.ipynb → PSI computation
h. Artifacts Storage
• Pickle models (.pkl), CSV scorecards, parquet scores
i. Presentation & Reporting
• Jupyter notebooks for interactive exploration
• reports/ folder for figures and summaries
j. Packaging & Deployment
• setup.py, requirements.txt for pip install
• Virtual environment management
Relationships & data flows
• Data Sources feed Data Ingestion → Cleaned Data
• Cleaned Data → EDA & Feature Engineering
• Preprocessed data → Model Training modules (PD, EAD, LGD)
• Trained models → Artifacts Storage
• Artifacts + Raw/Test data → Expected Loss computation → Output reports
• Year-t+1 data → Model Monitoring → PSI metrics
• src/ code is invoked by each notebook; notebooks orchestrate the flow.
Architectural patterns & design principles
• Layered pipeline (Ingestion → Processing → Modeling → Serving/Monitoring)
• CRISP-DM iterative framework
• Separation of concerns: notebooks for orchestration, src/ for reusable logic, artifacts for outputs
• Code modularity (utils, logger, exception handling)
• File-based storage for reproducibility
• Use of Jupyter for exploratory work, not production service
Relevant technologies & frameworks
• Python ecosystem: pandas, NumPy, scikit-learn, statsmodels
• Jupyter Notebooks for interactive development
• Parquet, CSV, pickle for data & model serialization
• Virtual environments (venv) for dependency isolation