# mikeo-ai/advanced-mlops-on-databricks-demo architecture

> Architecture diagram of the GitHub repository mikeo-ai/advanced-mlops-on-databricks-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mikeo-ai/advanced-mlops-on-databricks-demo
- Repository: https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo

## Overview

##### Project Type and Purpose
- This is a production-grade MLOps pipeline for automated model training, validation, deployment, batch inference, and monitoring on tabular data, built on top of Databricks MLOps Stacks.
- It provides end-to-end orchestration: feature engineering → model training → model validation → deployment → batch inference → monitoring, all managed by Databricks CLI bundles and GitHub Actions CI/CD.

##### File Structure Analysis
Top-level directories and their purposes:
- .github – GitHub Actions workflows for CI/CD (linting, testing, deploying ML resources).
- docs – Setup and usage guides (MLOps setup, PR guides, resource config).
- feature_engineering – Python modules + notebooks to compute and write features to Databricks Feature Store.
- training – Notebooks to train models and register them in MLflow.
- validation – Scripts & notebooks to evaluate model performance before deployment.
- monitoring – Code & notebooks to check production feature/model quality.
- deployment –
– batch_inference: code and notebooks to run scheduled batch predictions.
– model_deployment: scripts and notebooks to assign model aliases (promote to prod).
- resources – YAML definitions of Databricks jobs, models, feature-store artifacts, across dev/staging/prod.
- tests – Unit tests for feature modules and notebooks.
- cicd.tar.gz & .github/workflows/deploy-cicd.yml – Bundled CI/CD resources to bootstrap new projects.
- databricks.yml & project_params.json – Databricks CLI bundle entrypoint and parameter definitions.

##### README Insights
- The pipeline is defined as “bundles” loaded via the Databricks CLI, each bundle pointing to one of the YAML files under resources/.
- GitHub Actions workflows deploy these bundles per environment.
- Environments: dev, staging, prod, each with its own resource config.
- Core external dependencies: Databricks Workspace (with MLflow, Feature Store), cloud storage (Delta tables), GitHub.

a. Main Components
1. GitHub Repo
– Source code (feature_engineering, training, validation, monitoring, deployment)
– Infrastructure-as-Code (resources/*.yml, databricks.yml)
– CI/CD workflows (.github/workflows)
2. GitHub Actions CI/CD
– Lint & test code
– Build and deploy bundles to Databricks (dev → staging → prod)
3. Databricks CLI Bundles
– databricks.yml + resources/*.yml define jobs, schedules, MLflow models, feature tables
4. Databricks Workspace
– Feature Store (persistent feature tables)
– MLflow Model Registry (registered models, model versions, aliases)
– Jobs service (feature engineering jobs, training jobs, validation jobs, batch inference jobs, monitoring jobs)
5. Cloud Storage / Delta Lake
– Raw data tables (input)
– Feature tables (computed by feature_engineering)
– Prediction outputs (written by batch_inference)
6. Monitoring & Alerting
– Scheduled jobs to query feature/model metrics
– Violation detection scripts (monitoring/metric_violation_check_query.py)

b. Component Interactions & Data Flows
1. Developer pushes code → GitHub Actions triggers CI:
– runs unit tests → on success, triggers bundle deploy to target environment via Databricks CLI
2. Databricks workspace executes:
– Feature engineering job → writes features to Feature Store (Delta tables)
– Training job (notebook) → reads features → trains model → logs metrics to MLflow → registers model
– Validation job → reads registered model → runs validation notebooks → produces validation report
– Model deployment job → assigns alias (e.g., “prod”) to approved model version
– Batch inference job (scheduled) → loads model alias → reads fresh data → writes predictions back to storage
– Monitoring job (scheduled) → reads production data & predictions → computes quality metrics → triggers alerts if thresholds exceeded
3. Environments are parameterized via project_params.json and resources/*.yml to isolate dev/staging/prod.

c. Architectural Patterns & Principles
- Separation of concerns: code (ML logic) vs. infra (resource definitions) vs. CI/CD
- GitOps/MLOps: declarative resource config, version-controlled in Git
- Environment promotion: deploy same bundle to multiple environments with different configs
- Modular pipeline steps: feature engineering, training, validation, deployment, inference, monitoring

d. Key Technologies & Frameworks
- Python modules & Jupyter notebooks
- Databricks CLI Bundles (databricks.yml, resources/*.yml)
- GitHub Actions for CI/CD
- Databricks Feature Store & MLflow Model Registry
- Delta Lake on cloud storage
- pytest for unit tests

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD Layer
    subgraph "CI/CD Layer"
        style GitHubRepo fill:#D0E1F9,stroke:#000,stroke-width:1px
        GitHubRepo["GitHub Repository"]:::code
        style Workflows fill:#A8DADC,stroke:#000,stroke-width:1px
        Workflows["GitHub Actions Workflows"]:::ci
        style WorkflowsDocs fill:#A8DADC,stroke:#000,stroke-width:1px
        WorkflowsDocs["Workflows README"]:::ci
        style CICDBundle fill:#A8DADC,stroke:#000,stroke-width:1px
        CICDBundle["CI/CD Bundle Archive"]:::ci
        style CLIEntrypoint fill:#A8DADC,stroke:#000,stroke-width:1px
        CLIEntrypoint["Databricks CLI Entrypoint"]:::ci
        style Params fill:#A8DADC,stroke:#000,stroke-width:1px
        Params["CLI Parameter Definitions"]:::ci
        style Resources fill:#A8DADC,stroke:#000,stroke-width:1px
        Resources["Databricks Resources (YAML)"]:::ci
    end

    GitHubRepo -->|push code| Workflows
    Workflows -->|runs tests & lint| GitHubRepo
    Workflows -->|deploy bundles| CLIEntrypoint
    CLIEntrypoint --> Resources
    CLIEntrypoint --> Params
    CLIEntrypoint --> CICDBundle

    %% Databricks Workspace Layer
    subgraph "Databricks Workspace" 
        style FEJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        FEJob["Feature Engineering Job"]:::compute
        style TrainJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        TrainJob["Training Job"]:::compute
        style ValJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        ValJob["Validation Job"]:::compute
        style DeployJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        DeployJob["Model Deployment Job"]:::compute
        style BatchJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        BatchJob["Batch Inference Job"]:::compute
        style MonJob fill:#B7E4C7,stroke:#000,stroke-width:1px
        MonJob["Monitoring Job"]:::compute
        style FeatureStore fill:#AFE1AF,stroke:#000,stroke-width:1px
        FeatureStore["Feature Store (Delta)"]:::store
        style MLflowReg fill:#AFE1AF,stroke:#000,stroke-width:1px
        MLflowReg["MLflow Model Registry"]:::store
    end

    Resources --> FEJob
    Resources --> TrainJob
    Resources --> ValJob
    Resources --> DeployJob
    Resources --> BatchJob
    Resources --> MonJob

    FEJob -->|writes features| FeatureStore
    FeatureStore -->|reads features| TrainJob
    TrainJob -->|logs & registers model| MLflowReg
    MLflowReg -->|reads model| ValJob
    ValJob -->|validation report| MLflowReg
    MLflowReg -->|alias assign| DeployJob
    DeployJob -->|deploy alias| MLflowReg
    BatchJob -->|loads model alias| MLflowReg
    BatchJob -->|writes predictions| PredictionOutputs
    MonJob -->|reads data & preds| MonJob

    %% Data Layer
    subgraph "Data Layer"
        style RawData fill:#FFD6A5,stroke:#000,stroke-width:1px
        RawData["Raw Data Tables"]:::store
        style FeatureTables fill:#FFD6A5,stroke:#000,stroke-width:1px
        FeatureTables["Feature Tables"]:::store
        style PredictionOutputs fill:#FFD6A5,stroke:#000,stroke-width:1px
        PredictionOutputs["Prediction Outputs"]:::store
    end

    RawData --> FEJob
    BatchJob --> PredictionOutputs
    MonJob -->|alerts| GitHubRepo

    %% Code Modules for reference
    GitHubRepo --> FEModule
    GitHubRepo --> TrainModule
    GitHubRepo --> ValModule
    GitHubRepo --> BatchModule
    GitHubRepo --> DeployModule
    GitHubRepo --> MonModule
    GitHubRepo --> Docs
    GitHubRepo --> Tests

    subgraph "Feature Engineering Module"
        FEModule["feature_engineering/"]:::code
        dropoff["dropoff_features.py"]:::code
        pickup["pickup_features.py"]:::code
        feNb["GenerateAndWriteFeatures.py"]:::code
    end
    FEModule --> dropoff
    FEModule --> pickup
    FEModule --> feNb

    subgraph "Training Module"
        TrainModule["training/"]:::code
        trainNb["TrainWithFeatureStore.py"]:::code
    end
    TrainModule --> trainNb

    subgraph "Validation Module"
        ValModule["validation/"]:::code
        valScript["validation.py"]:::code
        valNb["ModelValidation.py"]:::code
    end
    ValModule --> valScript
    ValModule --> valNb

    subgraph "Batch Inference Module"
        BatchModule["deployment/batch_inference/"]:::code
        predictScript["predict.py"]:::code
        batchNb["BatchInference.py"]:::code
    end
    BatchModule --> predictScript
    BatchModule --> batchNb

    subgraph "Model Deployment Module"
        DeployModule["deployment/model_deployment/"]:::code
        deployScript["deploy.py"]:::code
        deployNb["ModelDeployment.py"]:::code
    end
    DeployModule --> deployScript
    DeployModule --> deployNb

    subgraph "Monitoring Module"
        MonModule["monitoring/"]:::code
        violation["metric_violation_check_query.py"]:::code
        monNb["MonitoredMetricViolationCheck.py"]:::code
    end
    MonModule --> violation
    MonModule --> monNb

    subgraph "Documentation"
        Docs["docs/"]:::docs
        mlopsGuide["mlops-setup.md"]:::docs
        prGuide["ml-pull-request.md"]:::docs
    end
    Docs --> mlopsGuide
    Docs --> prGuide

    subgraph "Tests"
        Tests["tests/"]:::test
        testDropoff["dropoff_features_test.py"]:::test
        testPickup["pickup_features_test.py"]:::test
        testTrain["test_notebooks.py"]:::test
    end
    Tests --> testDropoff
    Tests --> testPickup
    Tests --> testTrain

    %% Click events
    click Workflows "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/.github/workflows/"
    click WorkflowsDocs "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/.github/workflows/README.md"
    click CICDBundle "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/cicd.tar.gz"
    click CLIEntrypoint "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/databricks.yml"
    click Params "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/project_params.json"
    click Params "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/cicd_params.json"
    click Resources "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/resources/"
    click FEModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/feature_engineering/"
    click dropoff "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/feature_engineering/features/dropoff_features.py"
    click pickup "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/feature_engineering/features/pickup_features.py"
    click feNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/feature_engineering/notebooks/GenerateAndWriteFeatures.py"
    click TrainModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/training/"
    click trainNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/training/notebooks/TrainWithFeatureStore.py"
    click ValModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/validation/"
    click valScript "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/validation/validation.py"
    click valNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/validation/notebooks/ModelValidation.py"
    click BatchModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/deployment/batch_inference/"
    click predictScript "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/deployment/batch_inference/predict.py"
    click batchNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/deployment/batch_inference/notebooks/BatchInference.py"
    click DeployModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/deployment/model_deployment/"
    click deployScript "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/deployment/model_deployment/deploy.py"
    click deployNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/deployment/model_deployment/notebooks/ModelDeployment.py"
    click MonModule "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/monitoring/"
    click violation "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/monitoring/metric_violation_check_query.py"
    click monNb "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/monitoring/notebooks/MonitoredMetricViolationCheck.py"
    click Docs "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/docs/"
    click mlopsGuide "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/docs/mlops-setup.md"
    click prGuide "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/docs/ml-pull-request.md"
    click Tests "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/tree/main/tests/"
    click testDropoff "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/tests/feature_engineering/dropoff_features_test.py"
    click testPickup "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/tests/feature_engineering/pickup_features_test.py"
    click testTrain "https://github.com/mikeo-ai/advanced-mlops-on-databricks-demo/blob/main/tests/training/test_notebooks.py"

    %% Styles
    classDef ci fill:#A8DADC,stroke:#1D3557,color:#000
    classDef compute fill:#B7E4C7,stroke:#1D3557,color:#000
    classDef store fill:#FFD6A5,stroke:#1D3557,color:#000
    classDef code fill:#CED4DA,stroke:#1D3557,color:#000
    classDef docs fill:#FFD3B6,stroke:#1D3557,color:#000
    classDef test fill:#E5989B,stroke:#1D3557,color:#000
```
