# jostrm/azure-enterprise-scale-ml architecture

> Architecture diagram of the GitHub repository jostrm/azure-enterprise-scale-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jostrm/azure-enterprise-scale-ml
- Repository: https://github.com/jostrm/azure-enterprise-scale-ml

## Overview

##### Project type & purpose
– This is an open‑source “Enterprise Scale AI Factory” submodule/repo that provides:
- Infrastructure‑as‑Code templates (Bicep & Terraform) to bootstrap secure, enterprise‑scale AI landing zones and per‑project ESML (Enterprise Scale ML) or GenAI environments on Azure.
- DevOps pipeline definitions (.github/workflows, Azure DevOps, GitHub Actions) for CI/CD of landing zone and project deployments.
- Python SDKs (`esml`, `esmlrt`, `esmlfac`) and notebook templates for DataOps, MLOps, GenAIOps accelerators.
- Settings and sample code for user onboarding, RBAC/personas, network isolation (private endpoints, VWan, vNet/subnets), and data lake design.

##### File‑structure analysis
– Top‑level directories:
- `.github`, `azure_provisioning`, `environment_setup`→IaC & scripts for landing zones.
- `esml`, `esmlrt`, `esmlfac`→Python libraries/runtime interfaces & adapters.
- `notebook_templates`→prebuilt notebooks for Databricks, AML SDK v1/v2, GenAI.
- `settings`→enterprise‑wide & project‑specific JSON configs for compute, networking, automl, AKS, pipelines.
- CI/CD configs in `.github/workflows` and `azure_provisioning/azure_devops_pipelines`.

##### Key README insights
– Describes the two “project types” (ESML & GenAI-1) and three environments (Dev, Test, Prod) under a VWan or hub‑peered architecture.
– Emphasizes four main components:
1) Infra: AIFactory landing zones
2) Data lake template
3) DataOps/MLOps/GenAIOps templates
4) Accelerators (SDK, RAG chatbot, etc.)

a. Version‑control & bootstrap layer
- GitHub repo containing submodule
- Bootstrap scripts add Bicep/Terraform & notebook templates into user repo
b. CI/CD pipeline layer
- GitHub Actions / Azure DevOps build & release pipelines
c. Infrastructure provisioning layer
- Environment_setup: Bicep modules (esml-common, esml-genai, esml-project, esml-util), Terraform (optional)
- Creates: Resource Groups (Dev/Test/Prod), VWan or vNet & Subnets, Private Endpoints, Storage Accounts (Data Lake), Azure ML Workspaces, AKS, Container Registry, Databricks, Data Factory, Event Hubs, Key Vaults, App Insights, etc.
d. Configuration layer
- `settings` JSON for compute SKU, network ranges, pipeline templates, automl, AKS configs, RBAC persona groups
e. SDK & runtime layer
- Python packages (`esml`, `esmlrt`, `esmlfac`) providing:
– Interfaces to orchestrate pipelines in Azure ML / Databricks
– Helpers for data lake operations, model registration & scoring
f. Notebook & accelerator layer
- Notebook templates for quickstarts: DataOps ingestion, training/inference pipelines on AML SDK v1/v2, Databricks, Fabric
g. Users & personas
- CoreTeam (infra owners) vs ProjectTeam (data scientists & engineers)
- AAD security groups & RBAC for resource access

– Bootstrap → CI/CD triggers → IaC modules → Azure resource provisioning → Python SDK & notebooks run pipelines → Data & models flow into Data Lake, Model Registry, Batch/Online endpoints
– Settings feed into IaC modules and SDK at runtime
– Documentation drives user actions at each layer

##### Architectural patterns & design principles
– Layered, modular IaC: common, project, util modules (Bicep)
– Factory/multi‑project pattern: dynamic subnet/IP calculator, VWan/hub‑spoke, BYOvNet option
– Separation of concerns: infra vs pipelines vs runtime code
– Template‑driven: plug‑and‑play components for DataOps, MLOps, GenAIOps
– Security by default: private endpoints, vNet integration, RBAC personas

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Version Control & Bootstrap Layer
    subgraph "Version Control & Bootstrap" 
        GHRepo["GitHub Repo (AI Factory Submodule)"]:::github
        Bootstrap["Bootstrap Scripts & Templates"]:::bootstrap
    end

    %% CI/CD Layer
    subgraph "CI/CD Pipelines" 
        GHActions["GitHub Actions Workflows"]:::ci
        AzureDevOps["Azure DevOps Pipelines"]:::ci
    end

    %% Infrastructure Provisioning Layer
    subgraph "Infrastructure Provisioning" 
        InfraModules["Bicep Modules\n(esml-common, esml-genai-1, esml-project, esml-util)"]:::iac
        Terraform["Terraform Templates (optional)"]:::iac
        DataLakeModule["Data Lake Bicep Module"]:::iac
        Settings["Settings JSON\n(enterprise & project specific)"]:::config
    end

    %% Azure Landing Zones & Resources
    subgraph "Azure Landing Zones & Resources" 
        vNet["vNet & Subnets"]:::azure
        PrivateEndpoints["Private Endpoints"]:::azure
        DataLake["Data Lake Storage"]:::azure
        AMLWorkspace["Azure ML Workspace"]:::azure
        AKSCluster["AKS Cluster"]:::azure
        Databricks["Databricks"]:::azure
        KeyVault["Key Vault"]:::azure
        EventHubs["Event Hubs"]:::azure
        AppInsights["Application Insights"]:::azure
    end

    %% SDK & Runtime Layer
    subgraph "Python SDK & Runtime" 
        ESMLSDK["Python SDK - ESML"]:::sdk
        ESMLRT["Python SDK - ESMLRT"]:::sdk
        ESMLFAC["Python SDK - ESMLFAC Adapter"]:::sdk
    end

    %% Notebook Accelerators Layer
    subgraph "Notebook Accelerators" 
        NotebookTemplates["Notebook Templates (DataOps/MLOps/GenAIOps)"]:::notebook
    end

    %% Users & Personas
    subgraph "Users & Personas"
        CoreTeam["CoreTeam (Infra Owners)"]:::user
        ProjectTeam["ProjectTeam (Data Scientists & Engineers)"]:::user
    end

    %% Connections
    GHRepo -->|contains code| Bootstrap
    Bootstrap -->|push triggers| GHActions
    Bootstrap -->|push triggers| AzureDevOps

    GHActions -->|runs IaC modules| InfraModules
    AzureDevOps -->|runs IaC modules| InfraModules

    Settings -->|feeds configs| InfraModules
    Settings -->|feeds configs| ESMLSDK

    InfraModules -->|creates| vNet
    InfraModules -->|creates| PrivateEndpoints
    InfraModules -->|creates| DataLake
    InfraModules -->|creates| AMLWorkspace
    InfraModules -->|creates| AKSCluster
    InfraModules -->|creates| Databricks
    InfraModules -->|creates| KeyVault
    InfraModules -->|creates| EventHubs
    InfraModules -->|creates| AppInsights

    Terraform -->|alternative IaC| InfraModules

    ESMLSDK -->|orchestrates pipelines on| AMLWorkspace
    ESMLSDK -->|writes data to| DataLake
    ESMLSDK -->|registers models in| KeyVault

    ESMLRT -->|runtime helpers for| AMLWorkspace
    ESMLRT -->|runtime helpers for| AKSCluster

    ESMLFAC -->|fabric adapter for| Databricks

    NotebookTemplates -->|run on| Databricks
    NotebookTemplates -->|run on| AMLWorkspace

    CoreTeam -->|uses| GHRepo
    CoreTeam -->|reviews| Documentation
    ProjectTeam -->|uses| ESMLSDK
    ProjectTeam -->|uses| NotebookTemplates

    Documentation["Documentation & Diagrams"]:::doc

    Documentation -->|guides| CoreTeam
    Documentation -->|guides| ProjectTeam

    %% Click Events
    click InfraModules "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/environment_setup/aifactory/bicep"
    click Terraform "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/environment_setup/aifactory/terraform"
    click GHActions "https://github.com/jostrm/azure-enterprise-scale-ml/blob/main/.github/workflows/verify-aifactory-*.yml"
    click AzureDevOps "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/esml/azure_provisioning/azure_devops_pipelines"
    click DataLakeModule "https://github.com/jostrm/azure-enterprise-scale-ml/blob/main/environment_setup/aifactory/bicep/modules/dataLake.bicep"
    click Settings "https://github.com/jostrm/azure-enterprise-scale-ml/blob/main/settings/active_dev_test_prod.json"
    click ESMLSDK "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/esml/"
    click ESMLRT "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/esmlrt/"
    click ESMLFAC "https://github.com/jostrm/azure-enterprise-scale-ml/blob/main/esmlfac/adapter.py"
    click NotebookTemplates "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/notebook_templates/"
    click Bootstrap "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/bootstrap/"
    click Documentation "https://github.com/jostrm/azure-enterprise-scale-ml/tree/main/documentation/"

    %% Styles
    classDef github fill:#D0E8FF,stroke:#0A60FF
    classDef bootstrap fill:#E6F4EA,stroke:#28A745
    classDef ci fill:#CCE5FF,stroke:#007BFF
    classDef iac fill:#E2F0D9,stroke:#237841
    classDef config fill:#FFF4CC,stroke:#FFCC00
    classDef azure fill:#E0F7FA,stroke:#00838F
    classDef sdk fill:#FFE5CC,stroke:#FF6F00
    classDef notebook fill:#FFF3E0,stroke:#FB8C00
    classDef user fill:#FFF8E1,stroke:#F9A825
    classDef doc fill:#F3E5F5,stroke:#6A1B9A
```
