# azure-samples/modern-data-warehouse-dataops architecture

> Architecture diagram of the GitHub repository azure-samples/modern-data-warehouse-dataops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure-samples/modern-data-warehouse-dataops
- Repository: https://github.com/azure-samples/modern-data-warehouse-dataops

## Overview

##### Project Type and Purpose
- This is a “DataOps Samples” repository illustrating DevOps best practices for data engineering on Azure and Microsoft Fabric.
- It contains:
- Single-Technology Samples (e.g., CI/CD for ADF, Azure SQL, Stream Analytics, Fabric)
- End-to-End Samples (e.g., Medallion architecture with ADF + Databricks; Fabric)
- Utilities (scripts for Fabric automation)
- Key capabilities showcased: Infrastructure as Code, CI/CD pipelines, testing, observability/monitoring.

##### File-Structure Analysis
- Top-level directories:
- .github/workflows → GitHub Actions CI/CD definitions
- single_tech_samples → per-service sample folders (ADF, SQL, Stream Analytics, Fabric)
- e2e_samples → end-to-end pipeline reference implementations (parking_sensors, fabric_dataops_sample, etc.)
- utilities → stand-alone automation scripts
- Each sample follows a similar pattern:
- infrastructure (Bicep/Terraform/ARM templates)
- code or artifacts (notebooks, functions, pipelines)
- devops (Azure Pipelines YAML or GitHub Actions)
- tests (unit/integration)
- scripts (deploy, verify prerequisites)

##### README Insights
- The README groups content into “Single-Technology Samples” vs “End-to-End Samples” and lists major Azure/Fabric services.

a. Main Components
1. Developer Workflow
- Local clone of the repo
- Code editors / devcontainers
2. Source Code Repository
- GitHub (branches, PRs, templates)
3. CI/CD Orchestration
- GitHub Actions & Azure Pipelines (in .github/workflows + sample devops/ templates)
4. IaC Modules
- ARM/Bicep/Terraform definitions per sample
5. Provisioned Cloud Resources
- Azure Data Factory (pipelines, datasets, linked services)
- Azure Databricks (notebooks, clusters)
- Azure Synapse (serverless or dedicated SQL pools)
- Azure Stream Analytics
- Azure SQL Database
- Microsoft Fabric Lakehouse
- Supporting infra: Storage accounts, Key Vault, Log Analytics, Application Insights
6. Data Pipelines
- Ingest → Bronze → Silver → Gold stages (Medallion)
- Data-factory-triggered jobs, Databricks jobs
7. Testing
- Unit tests (sample code)
- Integration tests (data connectors)
8. Observability & Monitoring
- App Insights metrics
- Log Analytics workspaces
- Dashboards & workbooks
9. Utilities
- Fabric upload script
b. Component Interactions
- Testing feedback loops from “Tests” back to “CI/CD.”
c. Architectural Patterns & Principles
- Medallion (Bronze/Silver/Gold) pattern in the parking_sensors and fabric_dataops_sample e2e pipelines.
- Modular IaC for each service.
- Reusable CI/CD templates for multi-env deployments.
- Separation of concerns: infra, code, pipelines, tests, docs.
d. Key Technologies & Frameworks
- Azure: Data Factory, Databricks, Synapse, Stream Analytics, SQL DB, Key Vault, Storage, Monitor
- Microsoft Fabric Lakehouse
- IaC: Bicep, ARM, Terraform
- CI/CD: GitHub Actions, Azure Pipelines YAML templates
- Testing: pytest, MSTest/xUnit, Postman, custom data-connector scripts
- Devcontainers & Docker for local development

##### Diagramming Recommendations
- Use clear, rectangular nodes for logical components (Repo, CI/CD, IaC, Pipelines).
- Blue for Dev/Repo
- Green for CI/CD
- Orange for Infra
- Purple for Data Processing
- Teal for Monitoring

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Developer Environment"
        Dev["Developer<br/>Local Clone & Devcontainers"]:::dev
    end

    Dev -->|"Commit & PR"| Repo["GitHub Repository"]:::dev

    subgraph "CI/CD Orchestration"
        CI["GitHub Actions & Azure Pipelines"]:::cicd
    end

    Repo -->|"Trigger Workflows"| CI

    subgraph "IaC Modules"
        IaC1["parking_sensors IaC"]:::infra
        IaC2["fabric_dataops_sample Terraform"]:::infra
        IaC3["ADF CI/CD IaC"]:::infra
    end

    CI -->|"Plan & Apply"| IaC1
    CI -->|"Plan & Apply"| IaC2
    CI -->|"Plan & Apply"| IaC3

    subgraph "Provisioned Cloud Resources"
        ADF1["ADF - parking_sensors"]:::data
        ADF2["ADF - single_tech"]:::data
        Databricks1["Databricks - parking_sensors"]:::data
        Databricks2["Databricks - single_tech"]:::data
        Synapse["Synapse Workspace & SQL Pools"]:::data
        StreamAnalytics["Stream Analytics CI/CD"]:::data
        SQLDB["Azure SQL DB CI/CD"]:::data
        FabricLH["Fabric Lakehouse"]:::data
        InfraApps["App Insights & Log Analytics"]:::monitor
    end

    IaC1 -->|"Deploy ADF"| ADF1
    IaC3 -->|"Deploy ADF"| ADF2
    IaC1 -->|"Deploy Databricks"| Databricks1
    IaC3 -->|"Deploy Databricks"| Databricks2
    IaC1 -->|"Deploy Synapse & SQL"| Synapse
    IaC1 -->|"Deploy Observability"| InfraApps
    IaC2 -->|"Deploy Fabric"| FabricLH
    IaC3 -->|"Deploy Stream Analytics"| StreamAnalytics
    IaC3 -->|"Deploy SQL DB"| SQLDB

    subgraph "Data Pipelines (Medallion)"
        Bronze["Bronze Layer<br/>Raw Data on Storage"]:::data
        Silver["Silver Layer<br/>Processed Data"]:::data
        Gold["Gold Layer<br/>Aggregated Data"]:::data
    end

    ADF1 -->|"Ingest Raw"| Bronze
    Bronze -->|"Transform"| Databricks1
    Databricks1 -->|"Standardize"| Silver
    Silver -->|"Aggregate"| Databricks1
    Databricks1 -->|"Load"| Synapse

    subgraph "Testing"
        UT["Unit Tests"]:::dev
        IT["Integration Tests"]:::dev
    end

    CI -->|"Run Tests"| UT
    CI -->|"Run Tests"| IT

    InfraApps -->|"Collect Metrics"| Observ["Dashboards & Workbooks"]:::monitor

    subgraph "Utilities"
        Util1["Fabric Lakehouse Upload Script"]:::util
        Util2["Fabric Env Setup Script"]:::util
    end

    Util1 -->|"Upload"| FabricLH
    Util2 -->|"Setup"| FabricLH

    click CI "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/.github/workflows/"
    click IaC1 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/infrastructure/"
    click IaC2 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/fabric_dataops_sample/infrastructure/terraform/"
    click IaC3 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/datafactory/adf_cicd/infrastructure/"
    click ADF1 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/adf/"
    click ADF2 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/datafactory/adf_cicd/adf/"
    click Databricks1 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/databricks/notebooks/"
    click Databricks2 "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/databricks_ci_cd/notebook_jobs/"
    click Synapse "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/synapse/workspace/"
    click Synapse "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/sql/ddo_azuresqldw_dw/"
    click StreamAnalytics "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/streamanalytics/streamanalytics_ci_cd/"
    click SQLDB "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/azuresql/azuresql_ci_cd/"
    click FabricLH "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/fabric_dataops_sample/fabric/fabric_environment/"
    click FabricLH "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/fabric_dataops_sample/libraries/src/"
    click Bronze "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/adf/pipeline/P_Ingest_ParkingData.json"
    click Databricks1 "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/databricks/notebooks/02_standardize.py"
    click IT "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/parking_sensors/tests/integrationtests/"
    click UT "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/e2e_samples/fabric_dataops_sample/fabric/test/unit_tests/"
    click UT "https://github.com/azure-samples/modern-data-warehouse-dataops/tree/main/single_tech_samples/databricks_ci_cd/common/tests/"
    click InfraApps "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/infrastructure/modules/appinsights.bicep"
    click InfraApps "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/infrastructure/modules/log_analytics.bicep"
    click Util1 "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/utilities/fabric/lakehouse-file-upload/upload-file-to-lakehouse.py"
    click Util2 "https://github.com/azure-samples/modern-data-warehouse-dataops/blob/main/e2e_samples/fabric_dataops_sample/scripts/setup_fabric_environment.py"

    classDef dev fill:#CDEDF6,stroke:#1E90FF,color:#000
    classDef cicd fill:#DFF6DD,stroke:#008000,color:#000
    classDef infra fill:#FFE5B4,stroke:#FF8C00,color:#000
    classDef data fill:#EEDDFF,stroke:#800080,color:#000
    classDef monitor fill:#D0F0C0,stroke:#20B2AA,color:#000
    classDef util fill:#F0F0F0,stroke:#A9A9A9,color:#000
```
