How jostrm/azure-enterprise-scale-ml works
Project type & purpose
– This is an open‑source “Enterprise Scale AI Factory” submodule/repo that provides:
• Infrastructure‑as‑Code templates (Bicep & Terraform) to bootstrap secure, enterprise‑scale AI landing zones and per‑project ESML (Enterprise Scale ML) or GenAI environments on Azure.
• DevOps pipeline definitions (.github/workflows, Azure DevOps, GitHub Actions) for CI/CD of landing zone and project deployments.
• Python SDKs (esml, esmlrt, esmlfac) and notebook templates for DataOps, MLOps, GenAIOps accelerators.
• Settings and sample code for user onboarding, RBAC/personas, network isolation (private endpoints, VWan, vNet/subnets), and data lake design.
File‑structure analysis
– Top‑level directories:
• .github, azure_provisioning, environment_setup→IaC & scripts for landing zones.
• esml, esmlrt, esmlfac→Python libraries/runtime interfaces & adapters.
• notebook_templates→prebuilt notebooks for Databricks, AML SDK v1/v2, GenAI.
• settings→enterprise‑wide & project‑specific JSON configs for compute, networking, automl, AKS, pipelines.
• CI/CD configs in .github/workflows and azure_provisioning/azure_devops_pipelines.
Key README insights
– Describes the two “project types” (ESML & GenAI-1) and three environments (Dev, Test, Prod) under a VWan or hub‑peered architecture.
– Emphasizes four main components:
1) Infra: AIFactory landing zones
2) Data lake template
3) DataOps/MLOps/GenAIOps templates
4) Accelerators (SDK, RAG chatbot, etc.)
a. Version‑control & bootstrap layer
• GitHub repo containing submodule
• Bootstrap scripts add Bicep/Terraform & notebook templates into user repo
b. CI/CD pipeline layer
• GitHub Actions / Azure DevOps build & release pipelines
c. Infrastructure provisioning layer
• Environment_setup: Bicep modules (esml-common, esml-genai, esml-project, esml-util), Terraform (optional)
• Creates: Resource Groups (Dev/Test/Prod), VWan or vNet & Subnets, Private Endpoints, Storage Accounts (Data Lake), Azure ML Workspaces, AKS, Container Registry, Databricks, Data Factory, Event Hubs, Key Vaults, App Insights, etc.
d. Configuration layer
• settings JSON for compute SKU, network ranges, pipeline templates, automl, AKS configs, RBAC persona groups
e. SDK & runtime layer
• Python packages (esml, esmlrt, esmlfac) providing:
– Interfaces to orchestrate pipelines in Azure ML / Databricks
– Helpers for data lake operations, model registration & scoring
f. Notebook & accelerator layer
• Notebook templates for quickstarts: DataOps ingestion, training/inference pipelines on AML SDK v1/v2, Databricks, Fabric
g. Users & personas
• CoreTeam (infra owners) vs ProjectTeam (data scientists & engineers)
• AAD security groups & RBAC for resource access
– Bootstrap → CI/CD triggers → IaC modules → Azure resource provisioning → Python SDK & notebooks run pipelines → Data & models flow into Data Lake, Model Registry, Batch/Online endpoints
– Settings feed into IaC modules and SDK at runtime
– Documentation drives user actions at each layer
Architectural patterns & design principles
– Layered, modular IaC: common, project, util modules (Bicep)
– Factory/multi‑project pattern: dynamic subnet/IP calculator, VWan/hub‑spoke, BYOvNet option
– Separation of concerns: infra vs pipelines vs runtime code
– Template‑driven: plug‑and‑play components for DataOps, MLOps, GenAIOps
– Security by default: private endpoints, vNet integration, RBAC personas