# databeli/databricks_course architecture

> Architecture diagram of the GitHub repository databeli/databricks_course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/databeli/databricks_course
- Repository: https://github.com/databeli/databricks_course

## Overview

This repository is a collection of Databricks notebook examples, not one integrated application. Its principal recurring workflow is data ingestion, transformation across lakehouse layers, and analytics; separate examples demonstrate notebook use, jobs, streaming, AI, privacy, and data modeling. Source excerpts confirm selected implementations, while unsampled projects are represented only at the directory level; the empty README provides no additional architecture evidence.

Main entry points: [`18_Agentic_AI/02_Notebooks`](https://github.com/databeli/databricks_course/tree/main/18_Agentic_AI/02_Notebooks) (Agentic AI notebooks), [`05_Jobs/02_Job_Notebooks/01_Fetch_List_of_Tables.py`](https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/01_Fetch_List_of_Tables.py) (Table listing), [`11_Lakeflow_Declarative_Pipeline/my_transformation.py`](https://github.com/databeli/databricks_course/blob/main/11_Lakeflow_Declarative_Pipeline/my_transformation.py) (Declarative pipeline), [`01_Notebook_Features_Overview/01_Notebook_01_Main.py`](https://github.com/databeli/databricks_course/blob/main/01_Notebook_Features_Overview/01_Notebook_01_Main.py) (Notebook features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_notebooks["Notebook workflows"]
  node_notebook_features["Notebook features"]
  node_notebook_helpers["Notebook helpers"]
  node_notebook_run["Notebook execution"]
  node_jobs_setup["Job setup"]
  node_jobs_fetch["Table listing"]
  node_jobs_ingest["Table ingestion<br/>[02_Ingest_Data.py]"]
  node_jobs_transform["Job transformation"]
end

subgraph group_pipelines["Data pipelines"]
  node_auto_loader["Auto Loader<br/>[01_Auto_Loader.py]"]
  node_bank_bronze["Bank source to bronze"]
  node_bank_silver["Bank bronze to silver"]
  node_bank_gold["Bank gold driver"]
  node_pipeline_definition["Declarative pipeline"]
end

subgraph group_analytics["Analytics projects"]
  node_retail_bronze["Retail ingestion"]
  node_retail_silver["Retail silver transforms"]
  node_retail_gold["Retail sales facts<br/>[fact_sales.py]"]
  node_metric_view["Retail metrics<br/>[04_Metric View.py]"]
end

subgraph group_special["Specialized examples"]
  node_privacy["PII handling"]
  node_scd_models["SCD data models"]
  node_agentic_ai["Agentic AI notebooks"]
end

node_developer(("Notebook user"))
node_tables[("Catalog tables")]
node_file_sources["File inputs"]
node_vector_index[("Vector search index")]
node_ai_service{{"Databricks AI service"}}

node_developer -->|"uses"| node_notebook_features
node_notebook_features -.->|"runs via"| node_notebook_helpers
node_developer -->|"runs"| node_notebook_run
node_notebook_run -.->|"invokes"| node_jobs_fetch
node_jobs_setup -.->|"prepares"| node_jobs_fetch
node_jobs_fetch -.->|"reads"| node_tables
node_jobs_ingest -->|"reads and writes"| node_tables
node_jobs_transform -.->|"transforms"| node_tables
node_file_sources -->|"streams files"| node_auto_loader
node_auto_loader -->|"writes Delta"| node_tables
node_bank_bronze -->|"reads and writes"| node_tables
node_bank_silver -.->|"reads and writes"| node_tables
node_bank_gold -.->|"produces aggregates"| node_tables
node_retail_bronze -.->|"writes bronze"| node_tables
node_retail_silver -.->|"reads bronze; writes silver"| node_tables
node_retail_gold -.->|"reads silver; writes gold"| node_tables
node_metric_view -->|"defines metrics on"| node_tables
node_pipeline_definition -->|"reads streaming data"| node_file_sources
node_pipeline_definition -->|"defines tables and views"| node_tables
node_privacy -.->|"transforms PII"| node_tables
node_scd_models -.->|"models table history"| node_tables
node_developer -.->|"queries documents"| node_agentic_ai
node_agentic_ai -->|"searches"| node_vector_index
node_agentic_ai -.->|"requests completion"| node_ai_service

click node_notebook_features "https://github.com/databeli/databricks_course/blob/main/01_Notebook_Features_Overview/01_Notebook_01_Main.py"
click node_notebook_helpers "https://github.com/databeli/databricks_course/blob/main/01_Notebook_Features_Overview/01_Notebook_02_Helper.py"
click node_notebook_run "https://github.com/databeli/databricks_course/blob/main/02_Notebook_Run_from_Notebook/02_Notebook_01_Main.py"
click node_jobs_setup "https://github.com/databeli/databricks_course/blob/main/05_Jobs/01_Set_up_Notebooks/01_Setup_List_of_Tables.py"
click node_jobs_fetch "https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/01_Fetch_List_of_Tables.py"
click node_jobs_ingest "https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/02_Ingest_Data.py"
click node_jobs_transform "https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/03_Transform_Data.py"
click node_auto_loader "https://github.com/databeli/databricks_course/blob/main/06_Auto_Loader/01_Auto_Loader.py"
click node_bank_bronze "https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/02_Source_to_Silver/03_Source_to_Bronze.py"
click node_bank_silver "https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/02_Source_to_Silver/04_Bronze_to_Silver.py"
click node_bank_gold "https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/03_Silver_to_Gold/01_Silver_to_Gold_Driver.py"
click node_retail_bronze "https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/02_Notebooks/01_blob_to_bronze.py"
click node_retail_silver "https://github.com/databeli/databricks_course/tree/main/17_RetailQ_Project/03_ETL_Pipeline/bronze_to_silver"
click node_retail_gold "https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/03_ETL_Pipeline/silver_to_gold/fact_sales.py"
click node_metric_view "https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/02_Notebooks/04_Metric%20View.py"
click node_pipeline_definition "https://github.com/databeli/databricks_course/blob/main/11_Lakeflow_Declarative_Pipeline/my_transformation.py"
click node_privacy "https://github.com/databeli/databricks_course/tree/main/14_PII_Data_Handling"
click node_scd_models "https://github.com/databeli/databricks_course/tree/main/19_Data_Modelling/02_Code_Notebooks"
click node_agentic_ai "https://github.com/databeli/databricks_course/tree/main/18_Agentic_AI/02_Notebooks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_notebook_features,node_notebook_helpers,node_notebook_run,node_jobs_setup,node_jobs_fetch,node_jobs_ingest,node_jobs_transform,node_developer toneBlue
class node_auto_loader,node_bank_bronze,node_bank_silver,node_bank_gold,node_pipeline_definition,node_tables,node_vector_index toneAmber
class node_retail_bronze,node_retail_silver,node_retail_gold,node_metric_view toneMint
class node_privacy,node_scd_models,node_agentic_ai toneRose
class node_file_sources,node_ai_service toneIndigo
```

## Components

### Notebook workflows

- [Notebook features](https://github.com/databeli/databricks_course/blob/main/01_Notebook_Features_Overview/01_Notebook_01_Main.py): `01_Notebook_Features_Overview/01_Notebook_01_Main.py`
- [Notebook helpers](https://github.com/databeli/databricks_course/blob/main/01_Notebook_Features_Overview/01_Notebook_02_Helper.py): `01_Notebook_Features_Overview/01_Notebook_02_Helper.py`
- [Notebook execution](https://github.com/databeli/databricks_course/blob/main/02_Notebook_Run_from_Notebook/02_Notebook_01_Main.py): `02_Notebook_Run_from_Notebook/02_Notebook_01_Main.py`
- [Job setup](https://github.com/databeli/databricks_course/blob/main/05_Jobs/01_Set_up_Notebooks/01_Setup_List_of_Tables.py): `05_Jobs/01_Set_up_Notebooks/01_Setup_List_of_Tables.py`
- [Table listing](https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/01_Fetch_List_of_Tables.py): `05_Jobs/02_Job_Notebooks/01_Fetch_List_of_Tables.py`
- [Table ingestion](https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/02_Ingest_Data.py): `05_Jobs/02_Job_Notebooks/02_Ingest_Data.py`
- [Job transformation](https://github.com/databeli/databricks_course/blob/main/05_Jobs/02_Job_Notebooks/03_Transform_Data.py): `05_Jobs/02_Job_Notebooks/03_Transform_Data.py`

### Data pipelines

- [Auto Loader](https://github.com/databeli/databricks_course/blob/main/06_Auto_Loader/01_Auto_Loader.py): `06_Auto_Loader/01_Auto_Loader.py`
- [Bank source to bronze](https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/02_Source_to_Silver/03_Source_to_Bronze.py): `13_Banking_Project/02_Source_to_Silver/03_Source_to_Bronze.py`
- [Bank bronze to silver](https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/02_Source_to_Silver/04_Bronze_to_Silver.py): `13_Banking_Project/02_Source_to_Silver/04_Bronze_to_Silver.py`
- [Bank gold driver](https://github.com/databeli/databricks_course/blob/main/13_Banking_Project/03_Silver_to_Gold/01_Silver_to_Gold_Driver.py): `13_Banking_Project/03_Silver_to_Gold/01_Silver_to_Gold_Driver.py`
- [Declarative pipeline](https://github.com/databeli/databricks_course/blob/main/11_Lakeflow_Declarative_Pipeline/my_transformation.py): `11_Lakeflow_Declarative_Pipeline/my_transformation.py`

### Analytics projects

- [Retail ingestion](https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/02_Notebooks/01_blob_to_bronze.py): `17_RetailQ_Project/02_Notebooks/01_blob_to_bronze.py`
- [Retail silver transforms](https://github.com/databeli/databricks_course/tree/main/17_RetailQ_Project/03_ETL_Pipeline/bronze_to_silver): `17_RetailQ_Project/03_ETL_Pipeline/bronze_to_silver`
- [Retail sales facts](https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/03_ETL_Pipeline/silver_to_gold/fact_sales.py): `17_RetailQ_Project/03_ETL_Pipeline/silver_to_gold/fact_sales.py`
- [Retail metrics](https://github.com/databeli/databricks_course/blob/main/17_RetailQ_Project/02_Notebooks/04_Metric%20View.py): `17_RetailQ_Project/02_Notebooks/04_Metric View.py`

### Specialized examples

- [PII handling](https://github.com/databeli/databricks_course/tree/main/14_PII_Data_Handling): `14_PII_Data_Handling`
- [SCD data models](https://github.com/databeli/databricks_course/tree/main/19_Data_Modelling/02_Code_Notebooks): `19_Data_Modelling/02_Code_Notebooks`
- [Agentic AI notebooks](https://github.com/databeli/databricks_course/tree/main/18_Agentic_AI/02_Notebooks): `18_Agentic_AI/02_Notebooks`

### Other components

- **Notebook user**
- **Catalog tables**
- **File inputs**
- **Vector search index**
- **Databricks AI service**

## Connections

- Notebook user → Notebook features: uses
- Notebook features → Notebook helpers: runs via
- Notebook user → Notebook execution: runs
- Notebook execution → Table listing: invokes
- Job setup → Table listing: prepares
- Table listing → Catalog tables: reads
- Table ingestion → Catalog tables: reads and writes
- Job transformation → Catalog tables: transforms
- File inputs → Auto Loader: streams files
- Auto Loader → Catalog tables: writes Delta
- Bank source to bronze → Catalog tables: reads and writes
- Bank bronze to silver → Catalog tables: reads and writes
- Bank gold driver → Catalog tables: produces aggregates
- Retail ingestion → Catalog tables: writes bronze
- Retail silver transforms → Catalog tables: reads bronze; writes silver
- Retail sales facts → Catalog tables: reads silver; writes gold
- Retail metrics → Catalog tables: defines metrics on
- Declarative pipeline → File inputs: reads streaming data
- Declarative pipeline → Catalog tables: defines tables and views
- PII handling → Catalog tables: transforms PII
- SCD data models → Catalog tables: models table history
- Notebook user → Agentic AI notebooks: queries documents
- Agentic AI notebooks → Vector search index: searches
- Agentic AI notebooks → Databricks AI service: requests completion