# bishwarup307/bnp_paribas_cardiff_claim_management architecture

> Architecture diagram of the GitHub repository bishwarup307/bnp_paribas_cardiff_claim_management, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bishwarup307/bnp_paribas_cardiff_claim_management
- Repository: https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management

## Overview

##### Project Type and Purpose
- This is a data‐science/ML ensemble system built for the Kaggle “BNP Paribas Cardif Claims Management” competition.
- Its goal is to predict claim approval probability by stacking and blending many base learners in a three‐layer ensemble.

##### File Structure Analysis
- Top-level directories:
- ensemble/L1 – first-layer base learners (Python & R implementations)
- ensemble/L2 – second-layer meta-learner scripts
- standalone_models – single XGBoost variants
- Key patterns:
- A two‐stage stacking architecture (layer 1 generates out‐of‐fold predictions, layer 2 uses those as features)
- Scripts for feature subset generation (`generate_feature_subsets__.R`), model training, and prediction pipelines
- Mixed languages (Python, R, shell commands) indicating multiple runtime environments

##### Extracts from the README
- Three-layer ensemble design described (we see L1 and L2 folders; a third layer may be implicit or combined in L2)
- Models: XGBoost, neural nets, RF, h2o, Vowpal Wabbit, etc.
- Evaluation metric: binary crossentropy/log loss

a. Main Components
- Data Ingestion & Preprocessing – raw CSVs → feature engineering R/Python scripts
- Layer 1 Base Learners – each script in ensemble/L1 is a separate block (e.g., XGB, RF, NN, etc.)
- Stacking Engine – code that aggregates L1 out‐of‐fold predictions into a feature matrix
- Layer 2 Meta‐Learner – `calibrated_XGB_L2.py` that trains on stacked features
- Standalone Models – optional side‐by‐side XGB variants in standalone_models
- Evaluation & Submission – logloss computation, final CSV generation
- External Tools – H2O cluster, Keras/TensorFlow, Vowpal Wabbit CLI

b. Relationships & Data Flow
- Raw data flows into feature‐generation scripts → cleansed feature sets
- Features feed into each L1 model → produce cross‐validated prediction files
- Prediction files feed into the stacking engine → form new feature table for L2
- L2 meta‐learner consumes stacked features → outputs final probability scores
- Final results are evaluated (logloss) and written to submission file

c. Architectural Patterns
- Layered stacking (two explicit layers, implied third stacking step for final blend)
- Separation of concerns: each model implemented and tested in isolation
- Polyglot pipeline: orchestrator (could be a shell or Makefile) runs R and Python jobs

d. Key Technologies & Frameworks
- Languages: Python (scikit‐learn, sklearn wrappers, Keras), R (xgboost, randomForest), C++ tools
- Distributed engines: H2O for GBM/RF
- VW command‐line for Vowpal Wabbit
- Data formats: CSV, serialized model artifacts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Layer
    subgraph "1. Data Ingestion & Preprocessing"
        A1["generate_feature_subsets__.R"]:::dataPrep
        A2["generate_stacking_folds__.R"]:::dataPrep
        A3["stacked_feature_importance.R"]:::dataPrep
    end
    DP["Features (CSV)"]:::dataPrep

    A1 -->|"CSV"| DP
    A2 -->|"CSV"| DP
    A3 -->|"CSV"| DP

    %% Layer 1 Base Learners
    subgraph "2. Layer 1 Base Learners"
        B1["et_denormalized_layer_1.py"]:::layer1
        B2["et_layer_1.py"]:::layer1
        B3["h2o_gbm_v19_1.R"]:::layer1
        B4["h2o_rf_v19_1.R"]:::layer1
        B5["nn_denormalized_layer_1.py"]:::layer1
        B6["nn_v1_layer_1.py"]:::layer1
        B7["rf_layer_1.py"]:::layer1
        B8["vowpal_wabbit_layer_1.py"]:::layer1
        B9["xgb_denormalized_layer_1.py"]:::layer1
        B10["xgb_layer_1.R"]:::layer1
        B11["xgb_master_layer1.R"]:::layer1
        B12["xgb_stacked_v5.py"]:::layer1
    end

    DP -->|"Features"| B1
    DP -->|"Features"| B2
    DP -->|"Features"| B3
    DP -->|"Features"| B4
    DP -->|"Features"| B5
    DP -->|"Features"| B6
    DP -->|"Features"| B7
    DP -->|"Features"| B8
    DP -->|"Features"| B9
    DP -->|"Features"| B10
    DP -->|"Features"| B11
    DP -->|"Features"| B12

    %% Stacking Aggregator
    C["Stack Aggregator"]:::aggregator
    B1 -->|"Pkl"| C
    B2 -->|"Pkl"| C
    B3 -->|"RDS"| C
    B4 -->|"RDS"| C
    B5 -->|"Pkl"| C
    B6 -->|"Pkl"| C
    B7 -->|"Pkl"| C
    B8 -->|"VW Predictions"| C
    B9 -->|"Pkl"| C
    B10 -->|"RDS"| C
    B11 -->|"RDS"| C
    B12 -->|"Pkl"| C

    %% Layer 2 Meta-Learner
    subgraph "3. Layer 2 Meta-Learner"
        D1["calibrated_XGB_L2.py"]:::layer2
    end
    C -->|"Stacked Features (CSV)"| D1

    %% Standalone Models
    subgraph "Standalone Models"
        E1["xgb_binned.py"]:::standalone
        E2["xgb_denormalized.py"]:::standalone
    end
    DP -->|"Features"| E1
    DP -->|"Features"| E2

    E1 -->|"Pkl"| F
    E2 -->|"Pkl"| F

    %% Evaluation & Submission
    F["Evaluation & Submission"]:::eval
    D1 -->|"Predictions CSV"| F

    %% Click Events for component mapping
    click A1 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/generate_feature_subsets__.R"
    click A2 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/generate_stacking_folds__.R"
    click A3 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/stacked_feature_importance.R"
    click B1 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/et_denormalized_layer_1.py"
    click B2 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/et_layer_1.py"
    click B3 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/h2o_gbm_v19_1.R"
    click B4 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/h2o_rf_v19_1.R"
    click B5 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/nn_denormalized_layer_1.py"
    click B6 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/nn_v1_layer_1.py"
    click B7 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/rf_layer_1.py"
    click B8 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/vowpal_wabbit_layer_1.py"
    click B9 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/xgb_denormalized_layer_1.py"
    click B10 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/xgb_layer_1.R"
    click B11 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/xgb_master_layer1.R"
    click B12 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L1/xgb_stacked_v5.py"
    click D1 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/ensemble/L2/calibrated_XGB_L2.py"
    click E1 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/standalone_models/xgb_binned.py"
    click E2 "https://github.com/bishwarup307/bnp_paribas_cardiff_claim_management/blob/master/standalone_models/xgb_denormalized.py"

    %% Styles
    classDef dataPrep fill:#f39c12,stroke:#e67e22,color:#fff
    classDef layer1 fill:#3498db,stroke:#2980b9,color:#fff
    classDef aggregator fill:#9b59b6,stroke:#8e44ad,color:#fff
    classDef layer2 fill:#2ecc71,stroke:#27ae60,color:#fff
    classDef standalone fill:#5dade2,stroke:#3498db,color:#fff
    classDef eval fill:#7f8c8d,stroke:#95a5a6,color:#fff
```
