How benitomartin/mlops-databricks-credit-default works
• Data Sources & Ingestion:
• Raw dataset (data/data.csv).
• Volumes in Databricks (the “data” and “packages” volumes) that store data and the model wheel.
• Data Processing & Feature Engineering:
• Notebooks under “notebooks/create_source_data” for synthetic data generation.
• “notebooks/feature_engineering” for processing features and running MLflow experiments.
• Source-level modules (src/credit_default/data_cleaning.py, data_preprocessing.py and their Spark counterparts) which provide reusable functions.
• Model Training & Execution:
• Workflows in “workflows/train_model.py”, “evaluate_model.py”, etc.
• ML experiments orchestrated in notebooks (e.g., combined_mlflow_experiment_notebook.py, custom_mlflow_experiment_notebook.py).
• Model Serving & Feature Lookup:
• “notebooks/model_feature_serving” notebooks that handle A/B testing and endpoint serving.
• Monitoring & Alerting:
• Notebooks under “notebooks/monitoring” for setting up alerts, monitoring the Lakehouse, and triggering external requests.
• “bundle_monitoring.yml” and “databricks.yml” configurations that integrate monitoring within the Databricks environment.
• CI/CD and Deployment Pipeline:
• GitHub workflows under .github/workflows (cd.yml and ci.yml).
• Makefile commands and pre-commit configurations for code quality.
• Databricks CLI integration for deploying models (as reflected in the provided instructions).
• Auxiliary/Infrastructure:
• Tests under the “tests” folder to validate functionality.
• Project configuration files (project_config.yml, pyproject.toml) that define environment details.
Out the Data Flow and Interactions
• Model Deployment & Serving: Connect the training phase to deployment pipelines managed via the “workflows” folder. Illustrate the transformation from a trained model to production-ready services (with both A/B testing and feature lookup notebooks).
Architectural Patterns and Technologies
• Layered Architecture: Separate data ingestion, processing/feature engineering, training, serving, and monitoring as distinct layers.
• Hybrid Processing Approach: Note the presence of both local processing (via Jupyter notebooks and Python scripts) and distributed processing with Apache Spark (as signaled by the Spark-specific cleaning and preprocessing modules).
• Integration with External Platforms: Emphasize the integration with Databricks (via CLI, workflows, and asset bundle YAML files) and MLflow for experiment tracking.
• CI/CD and Automated Pipelines: Point out the continuous integration workflow files (.github/workflows) and Makefile utilities that ensure automation from code quality checks to deployment.
• Tools & Libraries: Clearly mention key technologies (Python, Pandas, NumPy, Scikit-learn, Apache Spark, MLflow, and Databricks) and how they interplay within respective architectural blocks.