How magaramol/loan-risk-classification-and-insights-etl-and-eda-pipeline works
Project Type and Purpose
• This is an end-to-end ETL (Extract, Transform, Load) and Exploratory Data Analysis (EDA) pipeline built in a Jupyter Notebook.
• Its goal is to ingest raw loan data, apply cleaning and feature engineering, classify borrowers by risk and tenure, recommend communication channels, produce visualizations, and persist results into a MySQL database.
Key Architectural Components
a. Data Source
• CSV file (dpd_zero.csv) stored locally or in cloud storage
• Potential for extension: additional file formats (Parquet, JSON), API endpoints
b. ETL Engine
• Python modules/functions orchestrated in a Jupyter Notebook
• Subcomponents:
– Extraction: pandas read_csv
– Transformation: cleaning, risk/tenure classification, segmentation, feature engineering
– Loading: SQLAlchemy + mysql.connector writing into MySQL
c. Analytics & Visualization
• Seaborn and Matplotlib components producing PNGs in the IMG folder
• Exploratory dashboards or notebooks for inspecting distributions
d. Storage Layer
• MySQL database: schema creation, table dpd, using SQLAlchemy engine
• External dependency: MySQL server (hosted or managed service)
e. Execution Environment
• Jupyter Notebook as the orchestration layer
• Python runtime with dependencies: pandas, seaborn, matplotlib, mysql-connector-python, pymysql, SQLAlchemy
Relationships & Data Flow
1. User/Notebook Kernel triggers pipeline
2. Extraction reads CSV → DataFrame
3. Transformation applies cleaning functions and feature engineering → enriched DataFrame
4a. Load step persists DataFrame into MySQL (via connector + SQLAlchemy)
4b. Visualization step generates charts → saved in IMG/ folder
5. Consumer (data analyst, BI tool) queries MySQL or inspects saved charts
Architectural Patterns & Design Principles
• Layered Architecture: separate extraction, transformation, loading, visualization layers
• Modular Functions: classify_risk, classify_tenure, recommend_channel, feature engineering helpers
• Single Source of Truth: all cleaned and engineered data persisted in MySQL
• Incremental Extension Points: adding new features, data sources, or visualization modules