# tybulonazure/dp-203 architecture

> Architecture diagram of the GitHub repository tybulonazure/dp-203, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tybulonazure/dp-203
- Repository: https://github.com/tybulonazure/dp-203

## Overview

This repository is a set of course materials for learning Azure data engineering, not a deployable data platform. Its principal user workflow is to follow the lessons from storage and ingestion through transformation and serving, with separate lessons on streaming and governance. The graph preserves these documented curriculum branches and illustrates a concrete Databricks-to-ADLS transformation example shown in source. Most other connections between lesson topics are not established by the sampled files, so they are not represented as runtime calls.

Main entry points: [`README.md`](https://github.com/tybulonazure/dp-203/blob/main/README.md) (Course materials), [`34 - Common data transformations/Common data transformations.py`](https://github.com/tybulonazure/dp-203/blob/main/34%20-%20Common%20data%20transformations/Common%20data%20transformations.py) (Batch transformations), [`35 - Writing data to ADLSg2/Writing data to ADLSg2.py`](https://github.com/tybulonazure/dp-203/blob/main/35%20-%20Writing%20data%20to%20ADLSg2/Writing%20data%20to%20ADLSg2.py) (Lake writes), [`09 - Data lake structure - Raw layer`](https://github.com/tybulonazure/dp-203/tree/main/09%20-%20Data%20lake%20structure%20-%20Raw%20layer) (Lake layers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_course["Course foundation"]
  node_course_materials["Course materials<br/>[README.md]"]
end

subgraph group_ingest["Storage and ingestion"]
  node_storage["Storage foundations"]
  node_formats["Data formats"]
  node_lake_layers["Lake layers"]
  node_adf["Data Factory"]
  node_adf_dynamic["Dynamic pipelines<br/>[pipeline.json]"]
  node_adf_runtime["Integration runtimes"]
  node_adf_errors["Pipeline error handling"]
  node_lake_security["Lake security"]
  node_adf_delivery["ADF deployment"]
end

subgraph group_process["Transform and serve"]
  node_dimensional["Dimensional modeling"]
  node_databricks["Databricks notebooks<br/>[Hello world!.py]"]
  node_adls_access["ADLS access"]
  node_transform["Batch transformations"]
  node_write_lake["Lake writes"]
  node_autoloader["Auto Loader<br/>[AutoLoader.py]"]
  node_synapse_spark["Synapse Spark<br/>[Notebook 2.py]"]
  node_data_flows["Data flows"]
  node_dedicated_sql[("Dedicated SQL pool<br/>[query.sql]")]
  node_sql_loading["SQL pool loading"]
  node_serverless_sql[("Serverless SQL pool")]
end

subgraph group_special["Extended capabilities"]
  node_logic_apps["Logic Apps"]
  node_synapse_pipelines["Synapse pipelines"]
  node_dbt["dbt"]
  node_streaming["Streaming ingestion"]
  node_stream_analytics["Stream Analytics"]
  node_purview["Data governance"]
end

node_learner(("Learner"))

node_learner -->|"follows"| node_course_materials
node_course_materials -->|"covers"| node_storage
node_course_materials -->|"covers"| node_formats
node_course_materials -->|"covers"| node_lake_layers
node_course_materials -->|"covers"| node_adf
node_course_materials -->|"covers"| node_databricks
node_course_materials -->|"covers"| node_dedicated_sql
node_course_materials -->|"covers"| node_streaming
node_course_materials -->|"covers"| node_purview
node_transform -->|"reads raw data"| node_adls_access
node_transform -->|"writes curated data"| node_write_lake
node_write_lake -->|"writes curated layer"| node_lake_layers
node_sql_loading -->|"writes and reads"| node_dedicated_sql

click node_course_materials "https://github.com/tybulonazure/dp-203/blob/main/README.md"
click node_storage "https://github.com/tybulonazure/dp-203/tree/main/03%20-%20Storage%20account%20overview"
click node_formats "https://github.com/tybulonazure/dp-203/tree/main/06%20-%20Common%20file%20types%20(CSV%2C%20XML%2C%20JSON)"
click node_lake_layers "https://github.com/tybulonazure/dp-203/tree/main/09%20-%20Data%20lake%20structure%20-%20Raw%20layer"
click node_adf "https://github.com/tybulonazure/dp-203/tree/main/10%20-%20Introduction%20to%20Azure%20Data%20Factory"
click node_adf_dynamic "https://github.com/tybulonazure/dp-203/blob/main/11%20-%20Dynamic%20ADF/pipeline.json"
click node_adf_runtime "https://github.com/tybulonazure/dp-203/tree/main/12%20-%20ADF%20integration%20runtimes%20(part%201)"
click node_adf_errors "https://github.com/tybulonazure/dp-203/tree/main/14%20-%20ADF%20error%20handling"
click node_lake_security "https://github.com/tybulonazure/dp-203/tree/main/17%20-%20Data%20lake%20security%20-%20anonymous%20access%2C%20access%20keys"
click node_adf_delivery "https://github.com/tybulonazure/dp-203/tree/main/22%20-%20ARM-based%20CICD%20for%20ADF%20(part%201)"
click node_dimensional "https://github.com/tybulonazure/dp-203/tree/main/29%20-%20Introduction%20to%20dimensional%20modelling"
click node_databricks "https://github.com/tybulonazure/dp-203/blob/main/32%20-%20A%20closer%20look%20at%20Databricks%20notebooks/Hello%20world!.py"
click node_adls_access "https://github.com/tybulonazure/dp-203/blob/main/33%20-%20Connecting%20to%20ADLSg2%20from%20Azure%20Databricks/Access%20via%20mount%20points.py"
click node_transform "https://github.com/tybulonazure/dp-203/blob/main/34%20-%20Common%20data%20transformations/Common%20data%20transformations.py"
click node_write_lake "https://github.com/tybulonazure/dp-203/blob/main/35%20-%20Writing%20data%20to%20ADLSg2/Writing%20data%20to%20ADLSg2.py"
click node_autoloader "https://github.com/tybulonazure/dp-203/blob/main/36%20-%20Azure%20Databricks%20Autoloader/AutoLoader.py"
click node_synapse_spark "https://github.com/tybulonazure/dp-203/blob/main/39%20-%20Azure%20Synapse%20Analytics%20-%20Spark%20Pools/Notebook%202.py"
click node_data_flows "https://github.com/tybulonazure/dp-203/tree/main/41%20-%20Transforming%20data%20with%20data%20flows"
click node_dedicated_sql "https://github.com/tybulonazure/dp-203/blob/main/43%20-%20Azure%20Synapse%20Analytics%20-%20Dedicated%20SQL%20Pool/query.sql"
click node_sql_loading "https://github.com/tybulonazure/dp-203/blob/main/44%20-%20Loading%20data%20to%20Dedicated%20SQL%20Pool/Save%20data%20to%20Synapse%20Dedicated%20SQL%20Pool%20-%20tests.py"
click node_serverless_sql "https://github.com/tybulonazure/dp-203/tree/main/47%20-%20Azure%20Synapse%20Analytics%20-%20Serverless%20SQL%20Pool"
click node_logic_apps "https://github.com/tybulonazure/dp-203/tree/main/26%20-%20Introduction%20to%20Logic%20Apps"
click node_synapse_pipelines "https://github.com/tybulonazure/dp-203/tree/main/27%20-%20Azure%20Synapse%20Analytics%20-%20Overview%2C%20Pipelines"
click node_dbt "https://github.com/tybulonazure/dp-203/tree/main/38%20-%20Introduction%20to%20dbt"
click node_streaming "https://github.com/tybulonazure/dp-203/tree/main/49%20-%20Introduction%20to%20streaming%20and%20Event%20Hubs"
click node_stream_analytics "https://github.com/tybulonazure/dp-203/blob/main/50%20-%20Stream%20Analytics%20overview/Stream%20analytics%20query.sql"
click node_purview "https://github.com/tybulonazure/dp-203/tree/main/52%20-%20Data%20governance%20with%20Microsoft%20Purview"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_course_materials toneBlue
class node_storage,node_formats,node_lake_layers,node_adf,node_adf_dynamic,node_adf_runtime,node_adf_errors,node_lake_security,node_adf_delivery toneAmber
class node_dimensional,node_databricks,node_adls_access,node_transform,node_write_lake,node_autoloader,node_synapse_spark,node_data_flows,node_dedicated_sql,node_sql_loading,node_serverless_sql toneMint
class node_logic_apps,node_synapse_pipelines,node_dbt,node_streaming,node_stream_analytics,node_purview toneRose
class node_learner toneIndigo
```

## Components

### Course foundation

- [Course materials](https://github.com/tybulonazure/dp-203/blob/main/README.md): `README.md`

### Storage and ingestion

- [Storage foundations](https://github.com/tybulonazure/dp-203/tree/main/03%20-%20Storage%20account%20overview): `03 - Storage account overview`
- [Data formats](https://github.com/tybulonazure/dp-203/tree/main/06%20-%20Common%20file%20types%20(CSV%2C%20XML%2C%20JSON)): `06 - Common file types (CSV, XML, JSON)`
- [Lake layers](https://github.com/tybulonazure/dp-203/tree/main/09%20-%20Data%20lake%20structure%20-%20Raw%20layer): `09 - Data lake structure - Raw layer`
- [Data Factory](https://github.com/tybulonazure/dp-203/tree/main/10%20-%20Introduction%20to%20Azure%20Data%20Factory): `10 - Introduction to Azure Data Factory`
- [Dynamic pipelines](https://github.com/tybulonazure/dp-203/blob/main/11%20-%20Dynamic%20ADF/pipeline.json): `11 - Dynamic ADF/pipeline.json`
- [Integration runtimes](https://github.com/tybulonazure/dp-203/tree/main/12%20-%20ADF%20integration%20runtimes%20(part%201)): `12 - ADF integration runtimes (part 1)`
- [Pipeline error handling](https://github.com/tybulonazure/dp-203/tree/main/14%20-%20ADF%20error%20handling): `14 - ADF error handling`
- [Lake security](https://github.com/tybulonazure/dp-203/tree/main/17%20-%20Data%20lake%20security%20-%20anonymous%20access%2C%20access%20keys): `17 - Data lake security - anonymous access, access keys`
- [ADF deployment](https://github.com/tybulonazure/dp-203/tree/main/22%20-%20ARM-based%20CICD%20for%20ADF%20(part%201)): `22 - ARM-based CICD for ADF (part 1)`

### Transform and serve

- [Dimensional modeling](https://github.com/tybulonazure/dp-203/tree/main/29%20-%20Introduction%20to%20dimensional%20modelling): `29 - Introduction to dimensional modelling`
- [Databricks notebooks](https://github.com/tybulonazure/dp-203/blob/main/32%20-%20A%20closer%20look%20at%20Databricks%20notebooks/Hello%20world!.py): `32 - A closer look at Databricks notebooks/Hello world!.py`
- [ADLS access](https://github.com/tybulonazure/dp-203/blob/main/33%20-%20Connecting%20to%20ADLSg2%20from%20Azure%20Databricks/Access%20via%20mount%20points.py): `33 - Connecting to ADLSg2 from Azure Databricks/Access via mount points.py`
- [Batch transformations](https://github.com/tybulonazure/dp-203/blob/main/34%20-%20Common%20data%20transformations/Common%20data%20transformations.py): `34 - Common data transformations/Common data transformations.py`
- [Lake writes](https://github.com/tybulonazure/dp-203/blob/main/35%20-%20Writing%20data%20to%20ADLSg2/Writing%20data%20to%20ADLSg2.py): `35 - Writing data to ADLSg2/Writing data to ADLSg2.py`
- [Auto Loader](https://github.com/tybulonazure/dp-203/blob/main/36%20-%20Azure%20Databricks%20Autoloader/AutoLoader.py): `36 - Azure Databricks Autoloader/AutoLoader.py`
- [Synapse Spark](https://github.com/tybulonazure/dp-203/blob/main/39%20-%20Azure%20Synapse%20Analytics%20-%20Spark%20Pools/Notebook%202.py): `39 - Azure Synapse Analytics - Spark Pools/Notebook 2.py`
- [Data flows](https://github.com/tybulonazure/dp-203/tree/main/41%20-%20Transforming%20data%20with%20data%20flows): `41 - Transforming data with data flows`
- [Dedicated SQL pool](https://github.com/tybulonazure/dp-203/blob/main/43%20-%20Azure%20Synapse%20Analytics%20-%20Dedicated%20SQL%20Pool/query.sql): `43 - Azure Synapse Analytics - Dedicated SQL Pool/query.sql`
- [SQL pool loading](https://github.com/tybulonazure/dp-203/blob/main/44%20-%20Loading%20data%20to%20Dedicated%20SQL%20Pool/Save%20data%20to%20Synapse%20Dedicated%20SQL%20Pool%20-%20tests.py): `44 - Loading data to Dedicated SQL Pool/Save data to Synapse Dedicated SQL Pool - tests.py`
- [Serverless SQL pool](https://github.com/tybulonazure/dp-203/tree/main/47%20-%20Azure%20Synapse%20Analytics%20-%20Serverless%20SQL%20Pool): `47 - Azure Synapse Analytics - Serverless SQL Pool`

### Extended capabilities

- [Logic Apps](https://github.com/tybulonazure/dp-203/tree/main/26%20-%20Introduction%20to%20Logic%20Apps): `26 - Introduction to Logic Apps`
- [Synapse pipelines](https://github.com/tybulonazure/dp-203/tree/main/27%20-%20Azure%20Synapse%20Analytics%20-%20Overview%2C%20Pipelines): `27 - Azure Synapse Analytics - Overview, Pipelines`
- [dbt](https://github.com/tybulonazure/dp-203/tree/main/38%20-%20Introduction%20to%20dbt): `38 - Introduction to dbt`
- [Streaming ingestion](https://github.com/tybulonazure/dp-203/tree/main/49%20-%20Introduction%20to%20streaming%20and%20Event%20Hubs): `49 - Introduction to streaming and Event Hubs`
- [Stream Analytics](https://github.com/tybulonazure/dp-203/blob/main/50%20-%20Stream%20Analytics%20overview/Stream%20analytics%20query.sql): `50 - Stream Analytics overview/Stream analytics query.sql`
- [Data governance](https://github.com/tybulonazure/dp-203/tree/main/52%20-%20Data%20governance%20with%20Microsoft%20Purview): `52 - Data governance with Microsoft Purview`

### Other components

- **Learner**

## Connections

- Learner → Course materials: follows (evidence: `README.md`)
- Course materials → Storage foundations: covers (evidence: `README.md`)
- Course materials → Data formats: covers (evidence: `README.md`)
- Course materials → Lake layers: covers (evidence: `README.md`)
- Course materials → Data Factory: covers (evidence: `README.md`)
- Course materials → Databricks notebooks: covers (evidence: `README.md`)
- Course materials → Dedicated SQL pool: covers (evidence: `README.md`)
- Course materials → Streaming ingestion: covers (evidence: `README.md`)
- Course materials → Data governance: covers (evidence: `README.md`)
- Batch transformations → ADLS access: reads raw data (evidence: `34 - Common data transformations/Common data transformations.py`)
- Batch transformations → Lake writes: writes curated data (evidence: `35 - Writing data to ADLSg2/Writing data to ADLSg2.py`)
- Lake writes → Lake layers: writes curated layer (evidence: `35 - Writing data to ADLSg2/Writing data to ADLSg2.py`)
- SQL pool loading → Dedicated SQL pool: writes and reads (evidence: `44 - Loading data to Dedicated SQL Pool/Save data to Synapse Dedicated SQL Pool - tests.py`)
