How end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35 works
Project Type & Purpose
• This is an end-to-end MLOps pipeline project for training, registering, deploying, and serving a hotel reservation classification model on Databricks. It includes data ingestion, feature engineering, model training/registration (MLflow), deployment (Databricks Serving), a Streamlit app for inference/monitoring, and CI/CD workflows.
Key Components
a. Data Sources & Storage
• External raw data (CSV, APIs) in data/raw
• Databricks Volume & Delta tables (ingestion outputs in data/interim & data/processed)
• Databricks Feature Store tables
b. Compute & Orchestration
• Databricks Jobs & Notebooks (process_data, train_register, deploy, monitoring)
• Databricks Asset Bundles (databricks.yml + resource templates)
• Local scripts & Taskfile tasks for development/demo
c. Model Lifecycle
• MLflow Tracking Server (experiments, runs, metrics, artifacts)
• MLflow Model Registry & Unity Catalog
• Model variants: basic, custom (hyperparameter tuning), feature-lookup
d. Serving & Inference
• Databricks Serving endpoint (Python handler in src/.../model_serving.py)
• Streamlit apps (app/app.py, app/app_monitoring.py)
• Functional test calls via HTTP
e. Monitoring & Visualization
• Metrics/Drift dashboards in Streamlit and notebooks (visualization/monitoring.py)
• MLflow Model Monitoring jobs
f. Dev & CI/CD Infrastructure
• Dev Environment: Devbox + UV + Taskfile
• CI/CD: GitHub Actions & GitLab CI pipelines (lint, test, build docs, deploy bundles)
• Multi-environment config (dev/acc/prd) via project_config.yml and .env
Interactions & Data Flow
1. Raw Data → scripts/00* or notebooks/run_upload_data.py → Databricks Volume
2. Volume Data → Data Processing Notebooks/Jobs → Delta Tables
3. Delta Tables → Feature Engineering (data_processor.py) → Feature Store
4. Feature Store → Model Training Notebooks/Jobs → MLflow Tracking → Registry
5. New Model Version → Conditional Databricks Job → Deploy Serving Endpoint
6. Client Apps (Streamlit or API tests) → Push inference requests → Serving Endpoint → Response
7. Serving Logs → Model Monitoring Jobs → Monitoring Dashboard
8. CI/CD triggers on Git events → Run tests, build docs, deploy asset bundles
Architectural Patterns & Principles
• Modular ML pipeline: clear separation of ingestion, preprocessing, feature engineering, training, serving
• Infrastructure as code: Taskfile, Databricks Asset Bundles, YAML templates, Devbox
• Multi-environment config & secret management
• Versioned model registry & feature store for reproducibility
• Automated CI/CD with comprehensive testing (unit, integration, functional)