How mikeo-ai/advanced-mlops-on-databricks-demo works
Project Type and Purpose
• This is a production-grade MLOps pipeline for automated model training, validation, deployment, batch inference, and monitoring on tabular data, built on top of Databricks MLOps Stacks.
• It provides end-to-end orchestration: feature engineering → model training → model validation → deployment → batch inference → monitoring, all managed by Databricks CLI bundles and GitHub Actions CI/CD.
File Structure Analysis
Top-level directories and their purposes:
• .github – GitHub Actions workflows for CI/CD (linting, testing, deploying ML resources).
• docs – Setup and usage guides (MLOps setup, PR guides, resource config).
• feature_engineering – Python modules + notebooks to compute and write features to Databricks Feature Store.
• training – Notebooks to train models and register them in MLflow.
• validation – Scripts & notebooks to evaluate model performance before deployment.
• monitoring – Code & notebooks to check production feature/model quality.
• deployment –
– batch_inference: code and notebooks to run scheduled batch predictions.
– model_deployment: scripts and notebooks to assign model aliases (promote to prod).
• resources – YAML definitions of Databricks jobs, models, feature-store artifacts, across dev/staging/prod.
• tests – Unit tests for feature modules and notebooks.
• cicd.tar.gz & .github/workflows/deploy-cicd.yml – Bundled CI/CD resources to bootstrap new projects.
• databricks.yml & project_params.json – Databricks CLI bundle entrypoint and parameter definitions.
README Insights
• The pipeline is defined as “bundles” loaded via the Databricks CLI, each bundle pointing to one of the YAML files under resources/.
• GitHub Actions workflows deploy these bundles per environment.
• Environments: dev, staging, prod, each with its own resource config.
• Core external dependencies: Databricks Workspace (with MLflow, Feature Store), cloud storage (Delta tables), GitHub.
a. Main Components
1. GitHub Repo
– Source code (feature_engineering, training, validation, monitoring, deployment)
– Infrastructure-as-Code (resources/*.yml, databricks.yml)
– CI/CD workflows (.github/workflows)
2. GitHub Actions CI/CD
– Lint & test code
– Build and deploy bundles to Databricks (dev → staging → prod)
3. Databricks CLI Bundles
– databricks.yml + resources/*.yml define jobs, schedules, MLflow models, feature tables
4. Databricks Workspace
– Feature Store (persistent feature tables)
– MLflow Model Registry (registered models, model versions, aliases)
– Jobs service (feature engineering jobs, training jobs, validation jobs, batch inference jobs, monitoring jobs)
5. Cloud Storage / Delta Lake
– Raw data tables (input)
– Feature tables (computed by feature_engineering)
– Prediction outputs (written by batch_inference)
6. Monitoring & Alerting
– Scheduled jobs to query feature/model metrics
– Violation detection scripts (monitoring/metric_violation_check_query.py)
b. Component Interactions & Data Flows
1. Developer pushes code → GitHub Actions triggers CI:
– runs unit tests → on success, triggers bundle deploy to target environment via Databricks CLI
2. Databricks workspace executes:
– Feature engineering job → writes features to Feature Store (Delta tables)
– Training job (notebook) → reads features → trains model → logs metrics to MLflow → registers model
– Validation job → reads registered model → runs validation notebooks → produces validation report
– Model deployment job → assigns alias (e.g., “prod”) to approved model version
– Batch inference job (scheduled) → loads model alias → reads fresh data → writes predictions back to storage
– Monitoring job (scheduled) → reads production data & predictions → computes quality metrics → triggers alerts if thresholds exceeded
3. Environments are parameterized via project_params.json and resources/*.yml to isolate dev/staging/prod.
c. Architectural Patterns & Principles
• Separation of concerns: code (ML logic) vs. infra (resource definitions) vs. CI/CD
• GitOps/MLOps: declarative resource config, version-controlled in Git
• Environment promotion: deploy same bundle to multiple environments with different configs
• Modular pipeline steps: feature engineering, training, validation, deployment, inference, monitoring
d. Key Technologies & Frameworks
• Python modules & Jupyter notebooks
• Databricks CLI Bundles (databricks.yml, resources/*.yml)
• GitHub Actions for CI/CD
• Databricks Feature Store & MLflow Model Registry
• Delta Lake on cloud storage
• pytest for unit tests