# gshelton/databricks architecture

> Architecture diagram of the GitHub repository gshelton/databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gshelton/databricks
- Repository: https://github.com/gshelton/databricks

## Overview

##### Project type and purpose
- This is neither a single‐page app nor a microservices system but a training-resource repository for the “Databricks Certified Data Engineer Associate” exam.
- It contains a logical progression of instructional assets (notebooks, SQL scripts, Python scripts) organized into five modules plus supporting “Includes” and “Labs” directories.

##### File structure
- Top-level directories:
- “1- Databricks Lakehouse Platform” (introductory notebooks and SQL demos on clusters, Delta tables, views)
- “2- ELT with Spark SQL and Python” (extract, load, transform examples)
- “3- Incremental Data Processing” (streaming, Auto Loader, multi-hop architectures)
- “4- Production Pipelines” (Delta Live Tables, orchestration tasks)
- “5- Data Governance” (permissions)
- “Includes” (setup scripts, dataset loaders)
- “Labs” and “Labs/Solutions” (hands-on exercises and solutions)
- Notice the pattern: each module folder mirrors a stage in a modern Databricks data-engineering workflow, from batch Delta operations through streaming and production pipelines, capped by governance.

##### Extract insights from README
- This repo is intended to be cloned into Databricks Repos and run as a self-paced course.

##### Components
a. Databricks Workspace
– Repos: houses this repository
– User Interface: Notebooks & Databricks SQL Editor
b. Compute Layer
– Clusters (interactive for notebooks, jobs clusters for pipelines)
– Delta Live Tables (declarative pipeline engine)
c. Storage Layer
– Delta Lake tables and databases on DBFS/ADLS/S3
– Auto Loader’s raw bronze storage location
d. Processing Stages (modules)
– Lakehouse Fundamentals (bronze/​silver/​gold zone demos)
– ELT jobs (SQL, Python transformations writing to Delta)
– Streaming/Incremental (Structured Streaming jobs, Auto Loader ingestion)
– Production Pipelines (Delta Live Tables definitions, orchestration tasks)
– Governance (Unity Catalog or ACLs on tables)
e. Supporting Scripts & Assets
– “Includes” for environment setup and shared helper code
– “Labs” for hands-on exercises, with “Solutions” side-by-side

##### Relationships and data flows
- Flow from Auto Loader bronze tables → transformations in ELT scripts (module 2) → silver/gold Delta tables.
- Delta Live Tables pipelines pulling from silver/gold tables and writing curated outputs.

##### Architectural patterns
- Medallion (bronze, silver, gold) layered data architecture
- Declarative pipeline (Delta Live Tables) vs. imperative notebooks
- Streaming ingestion → micro-batches (Structured Streaming + Auto Loader)
- Separation of concerns: raw ingestion, ETL/ELT, serving layer, governance

##### Final advice
- Be as granular as practical: break out clusters vs. pipelines, notebooks vs. SQL jobs, raw vs. curated zones.
- Resist over-crowding: group related notebooks/scripts into single “Lakehouse Fundamentals” or “Incremental Processing” blocks when needed.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Developer Workspace & Repos
    subgraph "Developer Workspace & Repos"
        WS_UI["Databricks UI\n(Notebooks & SQL Editor)"]:::ui
        WS_Repos["Repos (This Repo)"]:::repo
    end

    %% Ingestion & Bronze
    subgraph "Ingestion & Bronze Zone"
        AL["Auto Loader\n(Module 3)"]:::process
        Bronze["Bronze Delta Tables"]:::storage
    end

    %% Transformation & Silver/Gold
    subgraph "Transformation & Silver/Gold Zone"
        ELT["ELT Jobs\n(Module 2)"]:::process
        Silver["Silver Delta Tables"]:::storage
        Gold["Gold Delta Tables"]:::storage
    end

    %% Orchestration
    subgraph "Orchestration (Production)"
        DLT["Delta Live Tables Pipelines\n(Module 4)"]:::compute
        Curated["Curated Outputs"]:::storage
    end

    %% Governance
    subgraph "Governance & Security"
        Gov["Unity Catalog / ACLs\n(Module 5)"]:::gov
    end

    %% Supporting Scripts & Labs
    subgraph "Supporting Assets"
        Includes["Includes"]:::code
        Labs["Labs"]:::code
        LabsSol["Labs/Solutions"]:::code
    end

    %% Developer workspace relationships
    WS_Repos --> WS_UI
    WS_UI -->|runs notebooks| Module1
    WS_Repos --> Includes
    WS_Repos --> Labs
    WS_Repos --> LabsSol

    %% Data flow
    Source["External Source Data"]:::external --> AL
    AL --> Bronze
    Bronze --> ELT
    ELT --> Silver
    Silver --> Gold
    Gold --> DLT
    DLT --> Curated

    %% Interactive exploration
    Module1["Module 1: Lakehouse Fundamentals"]:::code -->|reads/writes| Bronze
    Module1 -->|reads/writes| Silver

    %% Governance wraps storage
    Bronze -.-> Gov
    Silver -.-> Gov
    Gold -.-> Gov
    Curated -.-> Gov

    %% Module blocks
    subgraph "Module 1 – Lakehouse Fundamentals" 
        direction TB
        M1_0["1.0 - Creating Clusters.py"]:::code
        M1_1["1.1 - Notebook Basics.py"]:::code
        M1_2["1.2 - Understanding Delta Tables.sql"]:::code
        M1_3["1.3 - Advanced Delta Lake Features.sql"]:::code
        M1_4["1.4 - Databases and Tables on Databricks.sql"]:::code
        M1_5A["1.5A - Views.sql"]:::code
        M1_5B["1.5B - Views (Session 2).sql"]:::code
    end

    subgraph "Module 2 – ELT with Spark SQL & Python"
        direction TB
        M2_1["2.1 - Querying Files.sql"]:::code
        M2_2["2.2 - Writing to Tables.sql"]:::code
        M2_3["2.3 - Advanced Transformations.sql"]:::code
        M2_4["2.4 - Higher Order Functions & SQL UDFs.sql"]:::code
    end

    subgraph "Module 3 – Incremental Data Processing"
        direction TB
        M3_1["3.1 - Structured Streaming.py"]:::code
        M3_2["3.2 - Auto Loader.py"]:::code
        M3_3["3.3 - Multi-Hop Architecture.py"]:::code
    end

    subgraph "Module 4 – Production Pipelines"
        direction TB
        M4_1["4.1 - Delta Live Tables.sql"]:::code
        M4_1_2["4.1.2 - Books Pipeline.sql"]:::code
        M4_2["4.2 - Pipeline Results.py"]:::code
        M4_3["4.3 - Land New Data Task.py"]:::code
    end

    subgraph "Module 5 – Data Governance"
        direction TB
        M5_1["5.1 - Managing Permissions.sql"]:::code
    end

    subgraph "Includes"
        direction TB
        INC1["Setup.py"]:::code
        INC2["Copy-Datasets.py"]:::code
        INC_IMG["images/"]:::code
    end

    subgraph "Labs"
        direction TB
        LABS_INC["Includes/Setup-Lab.py"]:::code
        LABS_IMG["Includes/images/"]:::code
    end

    subgraph "Lab Solutions"
        direction TB
        SOL_INC["Includes/Setup-Lab.py"]:::code
    end

    %% Click Events
    click M1_0 "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.0 - Creating Clusters.py"
    click M1_1 "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.1 - Notebook Basics.py"
    click M1_2 "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.2 - Understanding Delta Tables.sql"
    click M1_3 "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.3 - Advanced Delta Lake Features.sql"
    click M1_4 "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.4 - Databases and Tables on Databricks.sql"
    click M1_5A "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.5A - Views.sql"
    click M1_5B "https://github.com/gshelton/databricks/blob/main/1- Databricks Lakehouse Platform/1.5B - Views (Session 2).sql"

    click M2_1 "https://github.com/gshelton/databricks/blob/main/2- ELT with Spark SQL and Python/2.1 - Querying Files.sql"
    click M2_2 "https://github.com/gshelton/databricks/blob/main/2- ELT with Spark SQL and Python/2.2 - Writing to Tables.sql"
    click M2_3 "https://github.com/gshelton/databricks/blob/main/2- ELT with Spark SQL and Python/2.3 - Advanced Transformations.sql"
    click M2_4 "https://github.com/gshelton/databricks/blob/main/2- ELT with Spark SQL and Python/2.4 - Higher Order Functions and SQL UDFs.sql"

    click M3_1 "https://github.com/gshelton/databricks/blob/main/3- Incremental Data Processing/3.1 - Structured Streaming.py"
    click M3_2 "https://github.com/gshelton/databricks/blob/main/3- Incremental Data Processing/3.2 - Auto Loader.py"
    click M3_3 "https://github.com/gshelton/databricks/blob/main/3- Incremental Data Processing/3.3 - Multi-Hop Architecture.py"

    click M4_1 "https://github.com/gshelton/databricks/blob/main/4- Production Pipelines/4.1 - Delta Live Tables.sql"
    click M4_1_2 "https://github.com/gshelton/databricks/blob/main/4- Production Pipelines/4.1.2 - Books Pipeline.sql"
    click M4_2 "https://github.com/gshelton/databricks/blob/main/4- Production Pipelines/4.2 - Pipeline Results.py"
    click M4_3 "https://github.com/gshelton/databricks/blob/main/4- Production Pipelines/4.3 - Land New Data Task.py"

    click M5_1 "https://github.com/gshelton/databricks/blob/main/5- Data Governance/5.1 - Managing Permissions.sql"

    click INC1 "https://github.com/gshelton/databricks/blob/main/Includes/Setup.py"
    click INC2 "https://github.com/gshelton/databricks/blob/main/Includes/Copy-Datasets.py"
    click INC_IMG "https://github.com/gshelton/databricks/tree/main/Includes/images/"

    click LABS_INC "https://github.com/gshelton/databricks/blob/main/Labs/Includes/Setup-Lab.py"
    click LABS_IMG "https://github.com/gshelton/databricks/tree/main/Labs/Includes/images/"

    click SOL_INC "https://github.com/gshelton/databricks/blob/main/Labs/Solutions/Includes/Setup-Lab.py"

    %% Styles
    classDef ui fill:#E3F2FD,stroke:#64B5F6,stroke-width:2px
    classDef repo fill:#BBDEFB,stroke:#2196F3,stroke-width:2px
    classDef external fill:#F5F5F5,stroke:#90A4AE,stroke-width:1px,stroke-dasharray: 5 5
    classDef storage fill:#C8E6C9,stroke:#388E3C,stroke-width:2px
    classDef compute fill:#BBDEFB,stroke:#1976D2,stroke-width:2px
    classDef process fill:#FFECB3,stroke:#FFA000,stroke-width:2px
    classDef code fill:#FFE0B2,stroke:#FB8C00,stroke-width:2px
    classDef gov fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px
```
