# allmeidaapedro/lending-club-credit-scoring architecture

> Architecture diagram of the GitHub repository allmeidaapedro/lending-club-credit-scoring, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allmeidaapedro/lending-club-credit-scoring
- Repository: https://github.com/allmeidaapedro/lending-club-credit-scoring

## Overview

##### Project type and purpose
- This is an open-source machine-learning pipeline for credit risk modeling (“Lending Club Credit Scoring”).
- It implements the CRISP-DM process: business understanding ➔ data understanding & cleaning ➔ feature engineering ➔ modeling (PD, EAD, LGD) ➔ validation & scorecard generation ➔ model monitoring ➔ final policy.
- Artifacts (trained models, scorecards), raw inputs (parquet, Excel), Jupyter notebooks, utility code (src/), and visualization reports compose the project.

##### File structure
- Top level:
- artifacts/ – trained models & scorecards (PKL, CSV)
- input/ – raw & cleaned data (Excel, parquet)
- notebooks/ – exploratory, modeling, monitoring notebooks
- reports/ – static images (plots, logos)
- src/ – reusable Python modules (eda_utils, modelling_utils, logger, exception)
- setup.py, requirements.txt – packaging & dependencies
- Pattern: sequential pipeline driven by notebooks, with shared utility code in src/.

##### README for additional insights
- CRISP-DM six-step workflow
- Technologies: Python, pandas, NumPy, scikit-learn, statsmodels, Jupyter
- Outputs: PD scorecard, EAD & LGD models, expected loss calculator, credit policy rules, monitoring via PSI.

##### Main components
a. Data Sources
- External raw data on Kaggle (downloaded manually)
- input/LCDataDictionary.xlsx, input/clean_loan_data.parquet
b. Data Ingestion & Cleaning
- notebooks/1_data_cleaning_understanding.ipynb
- src/eda_utils.py, src/logger.py, src/exception.py
- Outputs cleaned_parquet → input/clean_loan_data.parquet
c. Exploratory Data Analysis
- notebooks/2_eda.ipynb
- src/eda_utils.py
- Generates plots → reports/
d. Feature Engineering & Preprocessing
- src/modelling_utils.py
- Dummy creation, discretization, encoding, scaling
e. Model Training
- PD Modeling (notebooks/3_pd_modeling.ipynb) → logistic regression + scorecard → artifacts/pd_model/scorecard.csv
- EAD & LGD Modeling (notebooks/4_lgd_ead_modeling.ipynb) → two-stage LGD, linear EAD → artifacts/ead_lgd_models/*.pkl
f. Expected Loss & Credit Policy Engine
- Notebook combines PD, EAD, LGD outputs → calculates EL = PD·EAD·LGD
- Risk classes & policy rules → reports/
g. Model Monitoring
- notebooks/5_pd_model_monitoring.ipynb → PSI computation
h. Artifacts Storage
- Pickle models (.pkl), CSV scorecards, parquet scores
i. Presentation & Reporting
- Jupyter notebooks for interactive exploration
- reports/ folder for figures and summaries
j. Packaging & Deployment
- setup.py, requirements.txt for pip install
- Virtual environment management

##### Relationships & data flows
- Data Sources feed Data Ingestion → Cleaned Data
- Cleaned Data → EDA & Feature Engineering
- Preprocessed data → Model Training modules (PD, EAD, LGD)
- Trained models → Artifacts Storage
- Artifacts + Raw/Test data → Expected Loss computation → Output reports
- Year-t+1 data → Model Monitoring → PSI metrics
- src/ code is invoked by each notebook; notebooks orchestrate the flow.

##### Architectural patterns & design principles
- Layered pipeline (Ingestion → Processing → Modeling → Serving/Monitoring)
- CRISP-DM iterative framework
- Separation of concerns: notebooks for orchestration, src/ for reusable logic, artifacts for outputs
- Code modularity (utils, logger, exception handling)
- File-based storage for reproducibility
- Use of Jupyter for exploratory work, not production service

##### Relevant technologies & frameworks
- Python ecosystem: pandas, NumPy, scikit-learn, statsmodels
- Jupyter Notebooks for interactive development
- Parquet, CSV, pickle for data & model serialization
- Virtual environments (venv) for dependency isolation

## Diagram (Mermaid)

```mermaid
graph TD
    %% Layer 1: Data Sources & Storage
    subgraph "Data Sources & Storage"
        classDef datastore fill:#caffbf,stroke:#2b9348
        DS1["LCDataDictionary.xlsx"]:::datastore
        DS2["clean_loan_data.parquet"]:::datastore
    end

    %% Layer 2: Data Ingestion & Cleaning
    subgraph "Data Ingestion & Cleaning"
        classDef code fill:#a2d2ff,stroke:#1d3557
        DCNB["1_data_cleaning_understanding.ipynb"]:::code
    end

    %% Shared Utilities
    subgraph "Shared Utilities (src/)"
        EDU["eda_utils.py"]:::code
        LOG["logger.py"]:::code
        EXC["exception.py"]:::code
        MU["modelling_utils.py"]:::code
    end

    %% Layer 3: Exploratory Data Analysis & Feature Engineering
    subgraph "EDA & Feature Engineering"
        classDef process fill:#ffe066,stroke:#ffad00
        EDA_NB["2_eda.ipynb"]:::process
    end

    %% Layer 4: Model Training
    subgraph "Model Training"
        subgraph "PD Modeling"
            PD_NB["3_pd_modeling.ipynb"]:::process
            SCORE["scorecard.csv"]:::datastore
            TRS["train_scores.parquet"]:::datastore
            TES["test_scores.parquet"]:::datastore
        end
        subgraph "EAD & LGD Modeling"
            LGD_NB["4_lgd_ead_modeling.ipynb"]:::process
            EADM["ead_model.pkl"]:::datastore
            LGDL["lgd_linear.pkl"]:::datastore
            LGDG["lgd_logistic.pkl"]:::datastore
        end
    end

    %% Layer 5: Expected Loss & Policy Engine
    subgraph "Expected Loss & Credit Policy Engine"
        EL["Expected Loss & Policy Engine"]:::process
    end

    %% Layer 6: Monitoring & Maintenance
    subgraph "Model Monitoring"
        MM_NB["5_pd_model_monitoring.ipynb"]:::process
    end

    %% Presentation & Reporting
    subgraph "Presentation & Reporting"
        classDef report fill:#ffadad,stroke:#d00000
        INB["All .ipynb Notebooks"]:::report
        RPTS["reports/ (static images)"]:::report
    end

    %% Packaging & Deployment
    subgraph "Packaging & Deployment"
        SETUP["setup.py"]:::code
        REQ["requirements.txt"]:::code
    end

    %% Data Flow Connections
    DS1 -->|raw data| DCNB
    DS2 -->|clean data input| EDA_NB
    DCNB -->|outputs cleaned parquet| DS2

    EDU --> DCNB
    LOG --> DCNB
    EXC --> DCNB

    DS2 -->|input for EDA| EDA_NB
    EDU --> EDA_NB

    EDA_NB -->|features| MU

    MU -->|preprocessed data| PD_NB
    MU -->|preprocessed data| LGD_NB

    PD_NB --> SCORE
    PD_NB --> TRS
    PD_NB --> TES

    LGD_NB --> EADM
    LGD_NB --> LGDL
    LGD_NB --> LGDG

    SCORE --> EL
    TRS --> EL
    TES --> EL
    EADM --> EL
    LGDL --> EL
    LGDG --> EL

    EL --> RPTS
    PD_NB --> INB
    LGD_NB --> INB

    MM_NB -->|year t+1 data| MM_NB
    DS2 --> MM_NB
    MM_NB --> RPTS

    SETUP --> INB
    REQ --> INB

    %% Shared Utilities usage
    EDU --> EDA_NB
    MU --> PD_NB
    MU --> LGD_NB
    EDU --> MM_NB

    %% Click Events
    click DS1 "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/input/LCDataDictionary.xlsx"
    click DS2 "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/input/clean_loan_data.parquet"
    click DCNB "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/notebooks/1_data_cleaning_understanding.ipynb"
    click EDU "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/src/eda_utils.py"
    click LOG "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/src/logger.py"
    click EXC "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/src/exception.py"
    click EDA_NB "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/notebooks/2_eda.ipynb"
    click MU "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/src/modelling_utils.py"
    click PD_NB "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/notebooks/3_pd_modeling.ipynb"
    click SCORE "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/pd_model/scorecard.csv"
    click TRS "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/pd_model/train_scores.parquet"
    click TES "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/pd_model/test_scores.parquet"
    click LGD_NB "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/notebooks/4_lgd_ead_modeling.ipynb"
    click EADM "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/ead_lgd_models/ead_model.pkl"
    click LGDL "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/ead_lgd_models/lgd_linear.pkl"
    click LGDG "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/artifacts/ead_lgd_models/lgd_logistic.pkl"
    click MM_NB "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/notebooks/5_pd_model_monitoring.ipynb"
    click RPTS "https://github.com/allmeidaapedro/lending-club-credit-scoring/tree/main/reports/"
    click INB "https://github.com/allmeidaapedro/lending-club-credit-scoring/tree/main/notebooks/"
    click SETUP "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/setup.py"
    click REQ "https://github.com/allmeidaapedro/lending-club-credit-scoring/blob/main/requirements.txt"
```
