# microsoftlearning/mslearn-databricks architecture

> Architecture diagram of the GitHub repository microsoftlearning/mslearn-databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoftlearning/mslearn-databricks
- Repository: https://github.com/microsoftlearning/mslearn-databricks

## Overview

##### Project type and purpose
- This is not a web or mobile app but a self-contained, hands-on “lab” repository for Microsoft Learn modules on Azure Databricks.
- It provides learners with:
- Instructions (Markdown under Instructions/Exercises)
- Sample data (CSV, JSON, Parquet under data/)
- Starter notebooks and configuration (sample_sales_notebook.py, .dbc, job-config.json)
- The goal is to teach Azure Databricks data-engineering, data-science, and AI patterns in a typical cloud environment.

##### File structure
- Top-level “Instructions/Exercises” directories group labs by domain (AI, DE, DS, LA).
- “data/” holds raw and edited datasets, spanning streaming, batch, notebooks, Delta Lake.
- Configuration artifacts include job-config.json, lab-acp.json, setup.ps1.
- No backend/frontend code – this is an infrastructure and orchestration exercise.

##### README for additional insights
- The README confirms that learners will spin up Azure resources (Databricks workspaces, clusters, storage) in their own subscription.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Environment
    subgraph "Local Environment" 
        direction TB
        Repo["Git Repository"]:::local
        Instructions["Instructions/Exercises"]:::local
        SampleData["Sample Data (data/)"]:::local
        NotebookPy["sample_sales_notebook.py"]:::local
        NotebookDBC["sample_sales_notebook.dbc"]:::local
        JobConfig["job-config.json"]:::local
        SetupPS["setup.ps1"]:::local
        LabACP["lab-acp.json"]:::local
        Readme["readme.md"]:::local
        Contrib[".github/CONTRIBUTING.md"]:::local
        IssueTemplate[".github/ISSUE_TEMPLATE.md"]:::local
        PRTemplate[".github/PULL_REQUEST_TEMPLATE.md"]:::local
    end

    %% Azure Cloud
    subgraph "Azure Subscription" 
        direction TB
        subgraph "Resource Group" 
            direction TB
            Storage["Storage Account (Blob/ADLS)"]:::azure
            subgraph "Databricks Workspace"
                direction TB
                Clusters["Compute Clusters"]:::compute
                NotebooksRepos["Notebooks & Repos"]:::azure
                MLflow["MLflow Model Registry"]:::azure
            end
            DataFactory["Data Factory / Jobs"]:::orchestrate
            EventHubs["Event Hubs (Streaming)"]:::azure
            Synapse["Synapse / SQL DW"]:::azure
        end
    end

    %% Flows
    SetupPS -.->|"provisions infra"| Storage
    SetupPS -.->|provisions infra| Clusters
    SetupPS -.->|provisions infra| DataFactory

    Storage -->|reads raw data| Clusters
    Clusters -->|writes Bronze/Silver/Gold| Storage
    NotebooksRepos -->|orchestrate notebooks| Clusters
    DataFactory -->|"schedules notebooks"| Clusters
    EventHubs -->|stream ingest| Storage
    Storage -->|serve Gold tables| Synapse
    Clusters -->|logs metrics| MLflow

    Instructions -->|guidance| Clusters
    SampleData -->|contains raw & edited| Storage
    NotebookPy -->|sample notebook| NotebooksRepos
    NotebookDBC -->|sample notebook| NotebooksRepos
    JobConfig -->|job definitions| DataFactory
    LabACP -->|deployment config| SetupPS

    %% Click Events
    click Instructions "https://github.com/microsoftlearning/mslearn-databricks/tree/main/Instructions/Exercises"
    click SampleData "https://github.com/microsoftlearning/mslearn-databricks/tree/main/data/"
    click NotebookPy "https://github.com/microsoftlearning/mslearn-databricks/blob/main/data/sample_sales_notebook.py"
    click NotebookDBC "https://github.com/microsoftlearning/mslearn-databricks/blob/main/data/sample_sales_notebook.dbc"
    click JobConfig "https://github.com/microsoftlearning/mslearn-databricks/blob/main/data/job-config.json"
    click SetupPS "https://github.com/microsoftlearning/mslearn-databricks/blob/main/setup.ps1"
    click LabACP "https://github.com/microsoftlearning/mslearn-databricks/blob/main/lab-acp.json"
    click Readme "https://github.com/microsoftlearning/mslearn-databricks/blob/main/readme.md"
    click Contrib "https://github.com/microsoftlearning/mslearn-databricks/blob/main/.github/CONTRIBUTING.md"
    click IssueTemplate "https://github.com/microsoftlearning/mslearn-databricks/blob/main/.github/ISSUE_TEMPLATE.md"
    click PRTemplate "https://github.com/microsoftlearning/mslearn-databricks/blob/main/.github/PULL_REQUEST_TEMPLATE.md"

    %% Styles
    classDef local fill:#f0f0f0,stroke:#999,stroke-width:1px
    classDef azure fill:#e0f7ff,stroke:#0288d1,stroke-width:1px
    classDef data fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
    classDef compute fill:#fff3e0,stroke:#ef6c00,stroke-width:1px
    classDef orchestrate fill:#f3e5f5,stroke:#6a1b9a,stroke-width:1px
```
