# joon-hub/lendingclubproject architecture

> Architecture diagram of the GitHub repository joon-hub/lendingclubproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joon-hub/lendingclubproject
- Repository: https://github.com/joon-hub/lendingclubproject

## Overview

##### Project Type and Purpose
- Type: Batch ETL/data-engineering pipeline for loan risk scoring.
- Purpose: Ingest raw loan/customer data, clean and validate it, compute component scores (payment history, defaults, financial health), combine into final loan scores, and write results & reports.

##### Main Architectural Layers & Components
a. Data Ingestion Layer
– Data Sources (CSV files): loan details, repayments, customer data, defaulters, delinquency.
– “data_cleaner.py” module with functions from utils/data_utils and utils/validation_utils.
– Spark session management via utils/spark_utils.
b. Data Processing Layer
– “data_processor.py”: transformations, deriving features.
– “loan_scorer.py”: scoring logic (PaymentHistory, DefaultsHistory, FinancialHealth, final aggregations).
– Configuration modules: config/paths_config.py (file paths), config/scoring_config.py (weights and thresholds).
c. Data Storage Layer
– Raw zone: data/raw/ → Parquet sink.
– Processed zone: data/processed/ → partitioned Parquet (by date).
– Output zone: loan_score/daily, loan_score/monthly (Parquet), loan_score/reports (CSV or Parquet).
d. Data Quality & Monitoring
– Validation functions in utils/validation_utils.
– Logging layer (log files directory).
– Data quality reports (can be represented as a side branch).
e. Orchestration & Runner
– “main.py” entrypoint or direct calls to loan_scorer.py with component flags.
f. Supporting Components
– Test suite (pytest tests/) – for completeness, represent as separate verification component.
– Setup/Dependencies: Pipfile / requirements.txt / setup.py.

##### Relationships & Data Flow
Data Sources → Ingestion → Cleaning & Validation → Processing → Scoring → Storage & Reports
- Utility modules (spark_utils, data_utils, validation_utils) feed into ingestion and processing.
- Tests reference core modules but do not affect production flow (place off to one side).

##### Architectural Patterns & Principles
- Layered architecture (Ingestion → Processing → Storage → Reporting).
- Modular design (separate Python modules for scoring, cleaning, utils, config).
- Config-driven parameters (paths_config, scoring_config).
- Use of Spark for parallel batch processing.
- Data partitioning and Parquet format for performance.

##### Technologies & Frameworks
- Python 3.x, Apache Spark (PySpark)
- Parquet file format
- pytest for unit tests
- configuration via Python modules, environment variables
- Logging for observability

##### Essential Elements to Capture
- All 3 scoring sub-components and their weightings (Payment History, Defaults, Financial Health).
- Paths: raw data directories, processed output, final loan_score output.
- Entry points (main.py or direct module calls).
- Utilities (Spark session, data and validation helpers).
- Config modules feeding parameters.
- Data formats at each step (CSV → Spark DataFrame → Parquet → Reports).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        DS1[/"Loan CSV Files"/]:::ingestion
        DS2[/"Repayments CSV Files"/]:::ingestion
        DS3[/"Customer CSV Files"/]:::ingestion
        DC["data_cleaner.py"]:::ingestion
    end

    %% Processing Layer
    subgraph "Processing Layer"
        DP["data_processor.py"]:::processing
        LS["loan_scorer.py"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        Raw["Raw Zone<br/>Parquet"]:::storage
        Processed["Processed Zone<br/>Partitioned Parquet"]:::storage
        Output["Output Zone<br/>Parquet"]:::storage
    end

    %% Reporting & Monitoring
    subgraph "Reporting/Monitoring"
        Reports["Reports Zone<br/>CSV/Parquet"]:::reporting
    end

    %% Orchestration
    subgraph "Orchestration"
        Main["main.py"]:::orchestration
    end

    %% Supporting Components
    subgraph "Supporting Components"
        SU["spark_utils.py"]:::utils
        DU["data_utils.py"]:::utils
        VU["validation_utils.py"]:::utils
        PC["paths_config.py"]:::config
        SC["scoring_config.py"]:::config
    end

    %% Tests
    subgraph "Tests & CI"
        T1["test_loan_scorer.py"]:::tests
        T2["test_utils.py"]:::tests
        T3["conftest.py"]:::tests
    end

    %% Data Flow
    DS1 -->|"CSV"| DC
    DS2 -->|"CSV"| DC
    DS3 -->|"CSV"| DC
    DC -->|"Parquet"| Raw
    Raw -->|"DataFrame"| DP
    DP -->|"Parquet"| Processed
    Processed -->|"DataFrame"| LS
    LS -->|"Parquet"| Output
    LS -->|"CSV/Parquet"| Reports

    %% Configuration & Utils References
    PC -.->|paths| DC
    PC -.->|paths| DP
    PC -.->|paths| LS
    SC -.->|weights/thresholds| LS
    SU -.->|SparkSession| DC
    SU -.->|SparkSession| DP
    DU -.->|helpers| DC
    VU -.->|validation| DC
    VU -.->|validation| DP

    %% Orchestration Flow
    Main --> DC
    Main --> DP
    Main --> LS

    %% Test Dependencies
    T1 -.->|tests scoring| LS
    T2 -.->|tests utils| DU
    T2 -.->|tests utils| VU
    T3 -.->|fixtures| DC
    T3 -.->|fixtures| DP
    T3 -.->|fixtures| LS

    %% Click Events
    click DC "https://github.com/joon-hub/lendingclubproject/blob/main/src/data_cleaner.py"
    click DU "https://github.com/joon-hub/lendingclubproject/blob/main/src/utils/data_utils.py"
    click VU "https://github.com/joon-hub/lendingclubproject/blob/main/src/utils/validation_utils.py"
    click SU "https://github.com/joon-hub/lendingclubproject/blob/main/src/utils/spark_utils.py"
    click DP "https://github.com/joon-hub/lendingclubproject/blob/main/src/data_processor.py"
    click LS "https://github.com/joon-hub/lendingclubproject/blob/main/src/loan_scorer.py"
    click PC "https://github.com/joon-hub/lendingclubproject/blob/main/src/config/paths_config.py"
    click SC "https://github.com/joon-hub/lendingclubproject/blob/main/src/config/scoring_config.py"
    click Main "https://github.com/joon-hub/lendingclubproject/blob/main/src/main.py"
    click T1 "https://github.com/joon-hub/lendingclubproject/blob/main/tests/test_loan_scorer.py"
    click T2 "https://github.com/joon-hub/lendingclubproject/blob/main/tests/test_utils.py"
    click T3 "https://github.com/joon-hub/lendingclubproject/blob/main/tests/conftest.py"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef reporting fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef orchestration fill:#87CEEB,stroke:#333,stroke-width:1px
    classDef utils fill:#F0F8FF,stroke:#333,stroke-width:1px
    classDef config fill:#F0E68C,stroke:#333,stroke-width:1px
    classDef tests fill:#D3D3D3,stroke:#333,stroke-width:1px
```
