How tybulonazure/dp-203 works
This repository is a set of course materials for learning Azure data engineering, not a deployable data platform. Its principal user workflow is to follow the lessons from storage and ingestion through transformation and serving, with separate lessons on streaming and governance. The graph preserves these documented curriculum branches and illustrates a concrete Databricks-to-ADLS transformation example shown in source. Most other connections between lesson topics are not established by the sampled files, so they are not represented as runtime calls.
Main entry points: README.md Course materials · Common data transformations.py Batch transformations · Writing data to ADLSg2.py Lake writes · 09 - Data lake structure - Raw layer Lake layers
Components 28
Course foundation
- Course materials
README.md
Storage and ingestion
- Storage foundations
03 - Storage account overview - Data formats
06 - Common file types (CSV, XML, JSON) - Lake layers
09 - Data lake structure - Raw layer - Data Factory
10 - Introduction to Azure Data Factory - Dynamic pipelines
11 - Dynamic ADF/pipeline.json - Integration runtimes
12 - ADF integration runtimes (part 1) - Pipeline error handling
14 - ADF error handling - Lake security
17 - Data lake security - anonymous access, access keys - ADF deployment
22 - ARM-based CICD for ADF (part 1)
Transform and serve
- Dimensional modeling
29 - Introduction to dimensional modelling - Databricks notebooks
32 - A closer look at Databricks notebooks/Hello world!.py - ADLS access
33 - Connecting to ADLSg2 from Azure Databricks/Access via mount points.py - Batch transformations
34 - Common data transformations/Common data transformations.py - Lake writes
35 - Writing data to ADLSg2/Writing data to ADLSg2.py - Auto Loader
36 - Azure Databricks Autoloader/AutoLoader.py - Synapse Spark
39 - Azure Synapse Analytics - Spark Pools/Notebook 2.py - Data flows
41 - Transforming data with data flows - Dedicated SQL pool
43 - Azure Synapse Analytics - Dedicated SQL Pool/query.sql - SQL pool loading
44 - Loading data to Dedicated SQL Pool/Save data to Synapse Dedicated SQL Pool - tests.py - Serverless SQL pool
47 - Azure Synapse Analytics - Serverless SQL Pool
Extended capabilities
- Logic Apps
26 - Introduction to Logic Apps - Synapse pipelines
27 - Azure Synapse Analytics - Overview, Pipelines - dbt
38 - Introduction to dbt - Streaming ingestion
49 - Introduction to streaming and Event Hubs - Stream Analytics
50 - Stream Analytics overview/Stream analytics query.sql - Data governance
52 - Data governance with Microsoft Purview
Other components
- Learner
Connections 13
- Learner → Course materials: follows
README.md - Course materials → Storage foundations: covers
README.md - Course materials → Data formats: covers
README.md - Course materials → Lake layers: covers
README.md - Course materials → Data Factory: covers
README.md - Course materials → Databricks notebooks: covers
README.md - Course materials → Dedicated SQL pool: covers
README.md - Course materials → Streaming ingestion: covers
README.md - Course materials → Data governance: covers
README.md - Batch transformations → ADLS access: reads raw data
34 - Common data transformations/Common data transformations.py - Batch transformations → Lake writes: writes curated data
35 - Writing data to ADLSg2/Writing data to ADLSg2.py - Lake writes → Lake layers: writes curated layer
35 - Writing data to ADLSg2/Writing data to ADLSg2.py - SQL pool loading → Dedicated SQL pool: writes and reads
44 - Loading data to Dedicated SQL Pool/Save data to Synapse Dedicated SQL Pool - tests.py