# benitomartin/mlops-databricks-credit-default architecture

> Architecture diagram of the GitHub repository benitomartin/mlops-databricks-credit-default, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/benitomartin/mlops-databricks-credit-default
- Repository: https://github.com/benitomartin/mlops-databricks-credit-default

## Overview

- Data Sources & Ingestion:
- Raw dataset (data/data.csv).
- Volumes in Databricks (the “data” and “packages” volumes) that store data and the model wheel.
- Data Processing & Feature Engineering:
- Notebooks under “notebooks/create_source_data” for synthetic data generation.
- “notebooks/feature_engineering” for processing features and running MLflow experiments.
- Source-level modules (src/credit_default/data_cleaning.py, data_preprocessing.py and their Spark counterparts) which provide reusable functions.
- Model Training & Execution:
- Workflows in “workflows/train_model.py”, “evaluate_model.py”, etc.
- ML experiments orchestrated in notebooks (e.g., combined_mlflow_experiment_notebook.py, custom_mlflow_experiment_notebook.py).
- Model Serving & Feature Lookup:
- “notebooks/model_feature_serving” notebooks that handle A/B testing and endpoint serving.
- Monitoring & Alerting:
- Notebooks under “notebooks/monitoring” for setting up alerts, monitoring the Lakehouse, and triggering external requests.
- “bundle_monitoring.yml” and “databricks.yml” configurations that integrate monitoring within the Databricks environment.
- CI/CD and Deployment Pipeline:
- GitHub workflows under .github/workflows (cd.yml and ci.yml).
- Makefile commands and pre-commit configurations for code quality.
- Databricks CLI integration for deploying models (as reflected in the provided instructions).
- Auxiliary/Infrastructure:
- Tests under the “tests” folder to validate functionality.
- Project configuration files (project_config.yml, pyproject.toml) that define environment details.

##### Out the Data Flow and Interactions
- Model Deployment & Serving: Connect the training phase to deployment pipelines managed via the “workflows” folder. Illustrate the transformation from a trained model to production-ready services (with both A/B testing and feature lookup notebooks).

##### Architectural Patterns and Technologies
- Layered Architecture: Separate data ingestion, processing/feature engineering, training, serving, and monitoring as distinct layers.
- Hybrid Processing Approach: Note the presence of both local processing (via Jupyter notebooks and Python scripts) and distributed processing with Apache Spark (as signaled by the Spark-specific cleaning and preprocessing modules).
- Integration with External Platforms: Emphasize the integration with Databricks (via CLI, workflows, and asset bundle YAML files) and MLflow for experiment tracking.
- CI/CD and Automated Pipelines: Point out the continuous integration workflow files (.github/workflows) and Makefile utilities that ensure automation from code quality checks to deployment.
- Tools & Libraries: Clearly mention key technologies (Python, Pandas, NumPy, Scikit-learn, Apache Spark, MLflow, and Databricks) and how they interplay within respective architectural blocks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        DA["Raw Data"]:::ingestion
        DC["Databricks Config"]:::ingestion
        MB["Monitoring Bundle Config"]:::ingestion
    end

    %% Data Processing & Feature Engineering Layer
    subgraph "Data Processing & Feature Engineering Layer"
        SN["Synthetic Data Notebook"]:::processing
        BN["Basic MLflow Notebook"]:::processing
        CN["Combined MLflow Notebook"]:::processing
        UN["Custom MLflow Notebook"]:::processing
        FN["Feature MLflow Notebook"]:::processing
        PN["Prepare Data Notebook"]:::processing
        DCP["Data Cleaning (Python)"]:::processing
        DPP["Data Preprocessing (Python)"]:::processing
        DCS["Data Cleaning (Spark)"]:::processing
        DPS["Data Preprocessing (Spark)"]:::processing
        ETF["MLflow Experiment Tracking"]:::processing
    end

    %% Model Training & Execution Layer
    subgraph "Model Training & Execution Layer"
        TM["Train Model Workflow"]:::training
        EM["Evaluate Model Workflow"]:::training
    end

    %% Model Serving & Feature Lookup Layer
    subgraph "Model Serving & Feature Lookup Layer"
        AB["A/B Testing Serving Notebook"]:::serving
        FL["Feature Lookup Serving Notebook"]:::serving
        FS["Feature Serving Notebook"]:::serving
        MS["Model Serving Notebook"]:::serving
    end

    %% Monitoring & Alerting Layer
    subgraph "Monitoring & Alerting Layer"
        CA["Create Alert Notebook"]:::monitoring
        CI["Create Inference Data Notebook"]:::monitoring
        LM["Lakehouse Monitoring Notebook"]:::monitoring
        ER["Endpoint Request Notebook"]:::monitoring
    end

    %% CI/CD & Deployment Pipeline
    subgraph "CI/CD & Deployment Pipeline"
        CICD["CI/CD Pipeline"]:::cicd
        GHCI["GitHub CI Workflow"]:::cicd
        GHCD["GitHub CD Workflow"]:::cicd
        MF["Makefile"]:::cicd
    end

    %% Auxiliary/Infrastructure
    subgraph "Auxiliary/Infrastructure"
        TST["Tests"]:::aux
        PC["Project Config"]:::aux
        PPC["PyProject Config"]:::aux
    end

    %% Data Flow Connections
    %% Ingestion to Processing
    DA --> SN
    DC --> SN
    MB --> SN

    %% Processing layer connections (Notebooks and Modules converge to MLflow Tracking)
    SN --> BN
    BN --> CN
    CN --> UN
    UN --> FN
    FN --> PN
    PN --> ETF
    DCP --> ETF
    DPP --> ETF
    DCS --> ETF
    DPS --> ETF

    %% Processing to Training
    ETF --> TM
    TM --> EM

    %% Training to Serving
    TM --> AB
    TM --> FL
    TM --> FS
    TM --> MS

    %% Serving to Monitoring
    AB --> CA
    FL --> CI
    FS --> LM
    MS --> ER

    %% Feedback loop from Monitoring to Processing (to trigger retraining)
    ER --> ETF

    %% CI/CD influences
    CICD --- SN
    CICD --- TM
    CICD --- AB

    %% Auxiliary supports CI/CD
    PC --> CICD
    PPC --> CICD
    TST --> CICD

    %% Additional CI/CD workflow connections
    GHCI --> CICD
    GHCD --> CICD
    MF --> CICD

    %% Styles
    classDef ingestion fill:#d1f2eb,stroke:#2ecc71,stroke-width:2px;
    classDef processing fill:#d6eaf8,stroke:#3498db,stroke-width:2px;
    classDef training fill:#fadbd8,stroke:#e74c3c,stroke-width:2px;
    classDef serving fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef monitoring fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
    classDef cicd fill:#f2d7d5,stroke:#e67e22,stroke-width:2px;
    classDef aux fill:#ebdef0,stroke:#8e44ad,stroke-width:2px;

    %% Click Events
    click DA "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/data/data.csv"
    click DC "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/databricks.yml"
    click MB "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/bundle_monitoring.yml"
    click SN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/create_source_data/create_source_data_notebook.py"
    click BN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/feature_engineering/basic_mlflow_experiment_notebook.py"
    click CN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/feature_engineering/combined_mlflow_experiment_notebook.py"
    click UN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/feature_engineering/custom_mlflow_experiment_notebook.py"
    click FN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/feature_engineering/feature_mlflow_experiment_notebook.py"
    click PN "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/feature_engineering/prepare_data_notebook.py"
    click DCP "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/src/credit_default/data_cleaning.py"
    click DPP "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/src/credit_default/data_preprocessing.py"
    click DCS "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/src/credit_default/data_cleaning_spark.py"
    click DPS "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/src/credit_default/data_preprocessing_spark.py"
    click TM "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/workflows/train_model.py"
    click EM "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/workflows/evaluate_model.py"
    click AB "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/model_feature_serving/AB_test_model_serving_notebbok.py"
    click FL "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/model_feature_serving/model_serving_feat_lookup_notebook.py"
    click FS "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/model_feature_serving/feature_serving_notebook.py"
    click MS "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/model_feature_serving/model_serving_notebook.py"
    click CA "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/monitoring/create_alert.py"
    click CI "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/monitoring/create_inference_data.py"
    click LM "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/monitoring/lakehouse_monitoring.py"
    click ER "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/notebooks/monitoring/send_request_to_endpoint.py"
    click GHCI "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/.github/workflows/ci.yml"
    click GHCD "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/.github/workflows/cd.yml"
    click MF "https://github.com/benitomartin/mlops-databricks-credit-default/tree/main/Makefile"
    click TST "https://github.com/benitomartin/mlops-databricks-credit-default/tree/main/tests/"
    click PC "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/project_config.yml"
    click PPC "https://github.com/benitomartin/mlops-databricks-credit-default/blob/main/pyproject.toml"
```
