# end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35 architecture

> Architecture diagram of the GitHub repository end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35
- Repository: https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35

## Overview

##### Project Type & Purpose
- This is an end-to-end MLOps pipeline project for training, registering, deploying, and serving a hotel reservation classification model on Databricks. It includes data ingestion, feature engineering, model training/registration (MLflow), deployment (Databricks Serving), a Streamlit app for inference/monitoring, and CI/CD workflows.

##### Key Components
a. Data Sources & Storage
- External raw data (CSV, APIs) in data/raw
- Databricks Volume & Delta tables (ingestion outputs in data/interim & data/processed)
- Databricks Feature Store tables
b. Compute & Orchestration
- Databricks Jobs & Notebooks (process_data, train_register, deploy, monitoring)
- Databricks Asset Bundles (`databricks.yml` + resource templates)
- Local scripts & Taskfile tasks for development/demo
c. Model Lifecycle
- MLflow Tracking Server (experiments, runs, metrics, artifacts)
- MLflow Model Registry & Unity Catalog
- Model variants: basic, custom (hyperparameter tuning), feature-lookup
d. Serving & Inference
- Databricks Serving endpoint (Python handler in `src/.../model_serving.py`)
- Streamlit apps (`app/app.py`, `app/app_monitoring.py`)
- Functional test calls via HTTP
e. Monitoring & Visualization
- Metrics/Drift dashboards in Streamlit and notebooks (`visualization/monitoring.py`)
- MLflow Model Monitoring jobs
f. Dev & CI/CD Infrastructure
- Dev Environment: Devbox + UV + Taskfile
- CI/CD: GitHub Actions & GitLab CI pipelines (lint, test, build docs, deploy bundles)
- Multi-environment config (dev/acc/prd) via `project_config.yml` and `.env`

##### Interactions & Data Flow
1. Raw Data → `scripts/00*` or `notebooks/run_upload_data.py` → Databricks Volume
2. Volume Data → Data Processing Notebooks/Jobs → Delta Tables
3. Delta Tables → Feature Engineering (`data_processor.py`) → Feature Store
4. Feature Store → Model Training Notebooks/Jobs → MLflow Tracking → Registry
5. New Model Version → Conditional Databricks Job → Deploy Serving Endpoint
6. Client Apps (Streamlit or API tests) → Push inference requests → Serving Endpoint → Response
7. Serving Logs → Model Monitoring Jobs → Monitoring Dashboard
8. CI/CD triggers on Git events → Run tests, build docs, deploy asset bundles

##### Architectural Patterns & Principles
- Modular ML pipeline: clear separation of ingestion, preprocessing, feature engineering, training, serving
- Infrastructure as code: Taskfile, Databricks Asset Bundles, YAML templates, Devbox
- Multi-environment config & secret management
- Versioned model registry & feature store for reproducibility
- Automated CI/CD with comprehensive testing (unit, integration, functional)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Local Dev Environment
    subgraph "Local Dev Environment"
        DEV_SCRIPTS["Taskfile & Devbox\n(Taskfile.yml, devbox.json)"]:::compute
        DEV_SCRIPTS -->|runs| ING_SCRIPTS
    end

    %% Databricks Workspace
    subgraph "Databricks Workspace"
        direction TB
        RAW_DATA["External Raw Data\n(data/raw)"]:::data
        click RAW_DATA "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/tree/main/data/raw"

        ING_SCRIPTS["Ingestion Scripts\n(scripts/*.py & notebooks/utils/run_upload_data.py)"]:::compute
        click ING_SCRIPTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/scripts/00.process_initial_data.py"
        click ING_SCRIPTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/scripts/01.process_new_data.py"
        click ING_SCRIPTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/utils/run_upload_data.py"

        VOLUME["Databricks Volume"]:::data

        PROC_JOBS["Data Processing Jobs\n(notebooks/process_data.py,\nnotebooks/process_new_data.py)"]:::compute
        click PROC_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/process_data.py"
        click PROC_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/process_new_data.py"

        DELTA["Delta Tables\n(data/interim & data/processed)"]:::data
        click DELTA "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/tree/main/data/interim"
        click DELTA "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/tree/main/data/processed"

        FEAT_ENG["Feature Engineering\n(src/.../data_processor.py)"]:::compute
        click FEAT_ENG "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/feature/data_processor.py"

        FEATURE_STORE["Feature Store Tables"]:::data
        TRAIN_JOBS["Model Training Jobs\n(notebooks/train_register_*.py,\nscripts/02*register*.py)"]:::compute
        click TRAIN_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/train_register_basic_model.py"
        click TRAIN_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/train_register_custom_model.py"
        click TRAIN_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/train_register_fe_model.py"
        click TRAIN_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/scripts/02.train_register_model.py"
        click TRAIN_JOBS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/scripts/02.b.train_register_custom_model.py"

        MLFLOW_TRACKING["MLflow Tracking Server"]:::data
        click MLFLOW_TRACKING "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/utils/run_create_mlflow_workspace.py"
        click MLFLOW_TRACKING "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/utils/run_cleanup_mlflow_experiments.py"

        MODEL_REGISTRY["Model Registry & Unity Catalog"]:::registry

        DEPLOY_JOB["Deployment Job\n(databricks.yml + resources/*.yml)"]:::compute
        click DEPLOY_JOB "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/databricks.yml"

        SERVING["Databricks Serving Endpoint"]:::registry
        click SERVING "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/serving/model_serving.py"

        %% Flows
        RAW_DATA -->|CSV upload| ING_SCRIPTS -->|Databricks API| VOLUME
        VOLUME -->|Delta write| PROC_JOBS -->|writes| DELTA
        DELTA -->|Feature lookup| FEAT_ENG -->|registers| FEATURE_STORE
        FEATURE_STORE -->|training data| TRAIN_JOBS -->|"MLflow SDK"| MLFLOW_TRACKING
        MLFLOW_TRACKING -->|"registers"| MODEL_REGISTRY -->|deploys| DEPLOY_JOB -->|bundle deploy| SERVING
    end

    %% Client Apps
    CLIENT_APP["Streamlit Apps\n(app/app.py, app/app_monitoring.py)"]:::app
    click CLIENT_APP "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/app/app.py"
    click CLIENT_APP "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/app/app_monitoring.py"

    FUNC_TESTS["Functional Tests\n(tests/functional/*.py, example.http)"]:::app
    click FUNC_TESTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/tests/functional/basic_model_serving_call.py"
    click FUNC_TESTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/tests/functional/custom_model_serving_call.py"
    click FUNC_TESTS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/tests/functional/example.http"

    CLIENT_APP -->|HTTP request| SERVING
    FUNC_TESTS -->|HTTP call| SERVING

    %% Monitoring
    MON_JOBS["Model Monitoring Jobs"]:::compute
    MON_DASH["Monitoring Dashboard\n(src/.../monitoring.py,\nnotebooks/create_monitoring.py)"]:::app
    click MON_DASH "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/visualization/monitoring.py"
    click MON_DASH "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/create_monitoring.py"
    click MON_DASH "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/notebooks/refresh_monitoring.py"

    SERVING -->|logs| MON_JOBS -->|updates| MON_DASH

    %% CI/CD Pipeline
    subgraph "CI/CD Pipeline"
        direction TB
        CI["CI/CD Pipelines\n(.github/workflows, .gitlab-ci.yml)"]:::ci
        click CI "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/.github/workflows/ci.yml"
        click CI "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/.github/workflows/cd.yml"
        click CI "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/.gitlab/.gitlab-ci.yml"
        CI -->|runs tests, docs, deploy| DEPLOY_JOB
    end

    %% Configuration & Utils
    subgraph "Configuration & Utilities"
        ENV["Env Config & Loader\n(project_config.yml, .env.template)"]:::data
        click ENV "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/project_config.yml"
        click ENV "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/.env.template"
        click ENV "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/utils/env_loader.py"
        click ENV "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/utils/config.py"

        DB_UTILS["Databricks Utils\n(src/.../databricks_utils.py)"]:::compute
        click DB_UTILS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/utils/databricks_utils.py"
        click DB_UTILS "https://github.com/end-to-end-mlops-databricks-4/marvelous-databricks-course-malganis35/blob/main/src/hotel_reservation/data/databricks_utils.py"
    end

    ENV --> ING_SCRIPTS
    ENV --> TRAIN_JOBS
    ENV --> DEPLOY_JOB

    classDef data fill:#a8d5a2,stroke:#333,stroke-width:1px
    classDef compute fill:#add8e6,stroke:#333,stroke-width:1px
    classDef registry fill:#ffa500,stroke:#333,stroke-width:1px
    classDef app fill:#d8bfd8,stroke:#333,stroke-width:1px
    classDef ci fill:#d3d3d3,stroke:#333,stroke-width:1px
```
