How joon-hub/lendingclubproject works
Project Type and Purpose
• Type: Batch ETL/data-engineering pipeline for loan risk scoring.
• Purpose: Ingest raw loan/customer data, clean and validate it, compute component scores (payment history, defaults, financial health), combine into final loan scores, and write results & reports.
Main Architectural Layers & Components
a. Data Ingestion Layer
– Data Sources (CSV files): loan details, repayments, customer data, defaulters, delinquency.
– “data_cleaner.py” module with functions from utils/data_utils and utils/validation_utils.
– Spark session management via utils/spark_utils.
b. Data Processing Layer
– “data_processor.py”: transformations, deriving features.
– “loan_scorer.py”: scoring logic (PaymentHistory, DefaultsHistory, FinancialHealth, final aggregations).
– Configuration modules: config/paths_config.py (file paths), config/scoring_config.py (weights and thresholds).
c. Data Storage Layer
– Raw zone: data/raw/ → Parquet sink.
– Processed zone: data/processed/ → partitioned Parquet (by date).
– Output zone: loan_score/daily, loan_score/monthly (Parquet), loan_score/reports (CSV or Parquet).
d. Data Quality & Monitoring
– Validation functions in utils/validation_utils.
– Logging layer (log files directory).
– Data quality reports (can be represented as a side branch).
e. Orchestration & Runner
– “main.py” entrypoint or direct calls to loan_scorer.py with component flags.
f. Supporting Components
– Test suite (pytest tests/) – for completeness, represent as separate verification component.
– Setup/Dependencies: Pipfile / requirements.txt / setup.py.
Relationships & Data Flow
Data Sources → Ingestion → Cleaning & Validation → Processing → Scoring → Storage & Reports
• Utility modules (spark_utils, data_utils, validation_utils) feed into ingestion and processing.
• Tests reference core modules but do not affect production flow (place off to one side).
Architectural Patterns & Principles
• Layered architecture (Ingestion → Processing → Storage → Reporting).
• Modular design (separate Python modules for scoring, cleaning, utils, config).
• Config-driven parameters (paths_config, scoring_config).
• Use of Spark for parallel batch processing.
• Data partitioning and Parquet format for performance.
Technologies & Frameworks
• Python 3.x, Apache Spark (PySpark)
• Parquet file format
• pytest for unit tests
• configuration via Python modules, environment variables
• Logging for observability
Essential Elements to Capture
• All 3 scoring sub-components and their weightings (Payment History, Defaults, Financial Health).
• Paths: raw data directories, processed output, final loan_score output.
• Entry points (main.py or direct module calls).
• Utilities (Spark session, data and validation helpers).
• Config modules feeding parameters.
• Data formats at each step (CSV → Spark DataFrame → Parquet → Reports).