# derar-alhussein/oreilly-databricks-dea architecture

> Architecture diagram of the GitHub repository derar-alhussein/oreilly-databricks-dea, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/derar-alhussein/oreilly-databricks-dea
- Repository: https://github.com/derar-alhussein/oreilly-databricks-dea

## Overview

This repository is the source material for a Databricks data-engineering study guide, not a deployable application. Its principal learner workflow is to study Databricks fundamentals, relational and Delta data, transformations, incremental processing, production pipelines, SQL analytics, and governance. The notebooks demonstrate distinct data-engineering stages; the sampled medallion notebook shows raw-to-bronze ingestion, silver enrichment, and gold aggregation. The available excerpts support links to shared setup and between those medallion stages, but not connections among most unsampled chapter examples.

Main entry points: [`Includes/School-Setup.py`](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Includes/School-Setup.py) (School setup), [`Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py) (Bronze ingestion), [`Chapter 5 - Processing Incremental Data/5.1 - Structured Streaming.py`](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.1%20-%20Structured%20Streaming.py) (Structured Streaming), [`Chapter 5 - Processing Incremental Data/5.2 - Auto Loader.py`](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.2%20-%20Auto%20Loader.py) (Auto Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_foundations["Databricks foundations"]
  node_getting_started["Getting started"]
  node_school_setup["School setup<br/>[School-Setup.py]"]
end

subgraph group_data["Data and transformations"]
  node_delta_lake["Delta Lake"]
  node_relational_entities["Relational entities"]
  node_spark_transformations["Spark transformations"]
end

subgraph group_incremental["Incremental processing"]
  node_structured_streaming["Structured Streaming"]
  node_auto_loader["Auto Loader"]
  node_bronze_ingestion["Bronze ingestion"]
  node_silver_enrichment["Silver enrichment"]
  node_gold_aggregation["Gold aggregation"]
end

subgraph group_production["Production pipelines"]
  node_dlt_pipelines["Delta Live Tables"]
  node_cdc_pipeline["CDC pipeline"]
  node_data_landing["Data landing"]
end

subgraph group_analytics["Analytics and governance"]
  node_sql_analytics["SQL analytics"]
  node_data_governance["Data governance"]
end

node_learner(("Learner"))

node_learner -->|"studies"| node_getting_started
node_structured_streaming -->|"runs setup"| node_school_setup
node_auto_loader -->|"runs setup"| node_school_setup
node_bronze_ingestion -->|"runs setup"| node_school_setup
node_data_landing -->|"runs setup"| node_school_setup
node_school_setup -->|"loads new data"| node_bronze_ingestion
node_bronze_ingestion -->|"streams bronze"| node_silver_enrichment
node_silver_enrichment -->|"streams silver"| node_gold_aggregation

click node_getting_started "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%201%20-%20Getting%20Started%20with%20Databricks/Demo%20Notebook.ipynb"
click node_delta_lake "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%202%20-%20Managing%20Data%20with%20Delta%20Lake/2.1%20-%20Delta%20Lake.sql"
click node_relational_entities "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%203%20-%20Mastering%20Relational%20Entities%20in%20Databricks/3.1%20-%20Databases%20and%20Tables.sql"
click node_spark_transformations "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%204%20-%20Transforming%20Data%20with%20Apache%20Spark/4.3%20-%20Advanced%20Transformations.sql"
click node_structured_streaming "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.1%20-%20Structured%20Streaming.py"
click node_auto_loader "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.2%20-%20Auto%20Loader.py"
click node_bronze_ingestion "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py"
click node_silver_enrichment "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py"
click node_gold_aggregation "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py"
click node_school_setup "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Includes/School-Setup.py"
click node_dlt_pipelines "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.1%20-%20Delta%20Live%20Tables.sql"
click node_cdc_pipeline "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.4%20-%20CDC%20Pipeline.sql"
click node_data_landing "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.3%20-%20Land%20New%20Data.py"
click node_sql_analytics "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%207%20-%20Exploring%20Databricks%20SQL/Example%20SQL%20Query.sql"
click node_data_governance "https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%208%20-%20Implementing%20Data%20Governance/Unity%20Catalog%20in%20Action.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_getting_started,node_school_setup toneBlue
class node_delta_lake,node_relational_entities,node_spark_transformations toneAmber
class node_structured_streaming,node_auto_loader,node_bronze_ingestion,node_silver_enrichment,node_gold_aggregation toneMint
class node_dlt_pipelines,node_cdc_pipeline,node_data_landing toneRose
class node_sql_analytics,node_data_governance,node_learner toneIndigo
```

## Components

### Databricks foundations

- [Getting started](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%201%20-%20Getting%20Started%20with%20Databricks/Demo%20Notebook.ipynb): `Chapter 1 - Getting Started with Databricks/Demo Notebook.ipynb`
- [School setup](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Includes/School-Setup.py): `Includes/School-Setup.py`

### Data and transformations

- [Delta Lake](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%202%20-%20Managing%20Data%20with%20Delta%20Lake/2.1%20-%20Delta%20Lake.sql): `Chapter 2 - Managing Data with Delta Lake/2.1 - Delta Lake.sql`
- [Relational entities](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%203%20-%20Mastering%20Relational%20Entities%20in%20Databricks/3.1%20-%20Databases%20and%20Tables.sql): `Chapter 3 - Mastering Relational Entities in Databricks/3.1 - Databases and Tables.sql`
- [Spark transformations](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%204%20-%20Transforming%20Data%20with%20Apache%20Spark/4.3%20-%20Advanced%20Transformations.sql): `Chapter 4 - Transforming Data with Apache Spark/4.3 - Advanced Transformations.sql`

### Incremental processing

- [Structured Streaming](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.1%20-%20Structured%20Streaming.py): `Chapter 5 - Processing Incremental Data/5.1 - Structured Streaming.py`
- [Auto Loader](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.2%20-%20Auto%20Loader.py): `Chapter 5 - Processing Incremental Data/5.2 - Auto Loader.py`
- [Bronze ingestion](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py): `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`
- [Silver enrichment](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py): `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`
- [Gold aggregation](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%205%20-%20Processing%20Incremental%20Data/5.3%20-%20Medallion%20Architecture.py): `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`

### Production pipelines

- [Delta Live Tables](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.1%20-%20Delta%20Live%20Tables.sql): `Chapter 6 - Building Production Pipelines/6.1 - Delta Live Tables.sql`
- [CDC pipeline](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.4%20-%20CDC%20Pipeline.sql): `Chapter 6 - Building Production Pipelines/6.4 - CDC Pipeline.sql`
- [Data landing](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%206%20-%20Building%20Production%20Pipelines/6.3%20-%20Land%20New%20Data.py): `Chapter 6 - Building Production Pipelines/6.3 - Land New Data.py`

### Analytics and governance

- [SQL analytics](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%207%20-%20Exploring%20Databricks%20SQL/Example%20SQL%20Query.sql): `Chapter 7 - Exploring Databricks SQL/Example SQL Query.sql`
- [Data governance](https://github.com/derar-alhussein/oreilly-databricks-dea/blob/main/Chapter%208%20-%20Implementing%20Data%20Governance/Unity%20Catalog%20in%20Action.sql): `Chapter 8 - Implementing Data Governance/Unity Catalog in Action.sql`

### Other components

- **Learner**

## Connections

- Learner → Getting started: studies (evidence: `README.md`)
- Structured Streaming → School setup: runs setup (evidence: `Chapter 5 - Processing Incremental Data/5.1 - Structured Streaming.py`)
- Auto Loader → School setup: runs setup (evidence: `Chapter 5 - Processing Incremental Data/5.2 - Auto Loader.py`)
- Bronze ingestion → School setup: runs setup (evidence: `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`)
- Data landing → School setup: runs setup (evidence: `Chapter 6 - Building Production Pipelines/6.3 - Land New Data.py`)
- School setup → Bronze ingestion: loads new data (evidence: `Includes/School-Setup.py`)
- Bronze ingestion → Silver enrichment: streams bronze (evidence: `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`)
- Silver enrichment → Gold aggregation: streams silver (evidence: `Chapter 5 - Processing Incremental Data/5.3 - Medallion Architecture.py`)