How derar-alhussein/oreilly-databricks-dea works
This repository is the source material for a Databricks data-engineering study guide, not a deployable application. Its principal learner workflow is to study Databricks fundamentals, relational and Delta data, transformations, incremental processing, production pipelines, SQL analytics, and governance. The notebooks demonstrate distinct data-engineering stages; the sampled medallion notebook shows raw-to-bronze ingestion, silver enrichment, and gold aggregation. The available excerpts support links to shared setup and between those medallion stages, but not connections among most unsampled chapter examples.
Main entry points: School-Setup.py School setup · 5.3 - Medallion Architecture.py Bronze ingestion · 5.1 - Structured Streaming.py Structured Streaming · 5.2 - Auto Loader.py Auto Loader
Components 16
Databricks foundations
- Getting started
Chapter 1 - Getting Started with Databricks/Demo Notebook.ipynb - School setup
Includes/School-Setup.py
Data and transformations
- Delta Lake
Chapter 2 - Managing Data with Delta Lake/2.1 - Delta Lake.sql - Relational entities
Chapter 3 - Mastering Relational Entities in Databricks/3.1 - Databases and Tables.sql - Spark transformations
Chapter 4 - Transforming Data with Apache Spark/4.3 - Advanced Transformations.sql
Incremental processing
- Structured Streaming
Chapter 5 - Processing Incremental Data/5.1 - Structured Streaming.py - Auto Loader
Chapter 5 - Processing Incremental Data/5.2 - Auto Loader.py - Bronze ingestion
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py - Silver enrichment
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py - Gold aggregation
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py
Production pipelines
- Delta Live Tables
Chapter 6 - Building Production Pipelines/6.1 - Delta Live Tables.sql - CDC pipeline
Chapter 6 - Building Production Pipelines/6.4 - CDC Pipeline.sql - Data landing
Chapter 6 - Building Production Pipelines/6.3 - Land New Data.py
Analytics and governance
- SQL analytics
Chapter 7 - Exploring Databricks SQL/Example SQL Query.sql - Data governance
Chapter 8 - Implementing Data Governance/Unity Catalog in Action.sql
Other components
- Learner
Connections 8
- Learner → Getting started: studies
README.md - Structured Streaming → School setup: runs setup
Chapter 5 - Processing Incremental Data/5.1 - Structured Streaming.py - Auto Loader → School setup: runs setup
Chapter 5 - Processing Incremental Data/5.2 - Auto Loader.py - Bronze ingestion → School setup: runs setup
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py - Data landing → School setup: runs setup
Chapter 6 - Building Production Pipelines/6.3 - Land New Data.py - School setup → Bronze ingestion: loads new data
Includes/School-Setup.py - Bronze ingestion → Silver enrichment: streams bronze
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py - Silver enrichment → Gold aggregation: streams silver
Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py