How gshelton/databricks works
Project type and purpose
• This is neither a single‐page app nor a microservices system but a training-resource repository for the “Databricks Certified Data Engineer Associate” exam.
• It contains a logical progression of instructional assets (notebooks, SQL scripts, Python scripts) organized into five modules plus supporting “Includes” and “Labs” directories.
File structure
• Top-level directories:
• “1- Databricks Lakehouse Platform” (introductory notebooks and SQL demos on clusters, Delta tables, views)
• “2- ELT with Spark SQL and Python” (extract, load, transform examples)
• “3- Incremental Data Processing” (streaming, Auto Loader, multi-hop architectures)
• “4- Production Pipelines” (Delta Live Tables, orchestration tasks)
• “5- Data Governance” (permissions)
• “Includes” (setup scripts, dataset loaders)
• “Labs” and “Labs/Solutions” (hands-on exercises and solutions)
• Notice the pattern: each module folder mirrors a stage in a modern Databricks data-engineering workflow, from batch Delta operations through streaming and production pipelines, capped by governance.
Extract insights from README
• This repo is intended to be cloned into Databricks Repos and run as a self-paced course.
Components
a. Databricks Workspace
– Repos: houses this repository
– User Interface: Notebooks & Databricks SQL Editor
b. Compute Layer
– Clusters (interactive for notebooks, jobs clusters for pipelines)
– Delta Live Tables (declarative pipeline engine)
c. Storage Layer
– Delta Lake tables and databases on DBFS/ADLS/S3
– Auto Loader’s raw bronze storage location
d. Processing Stages (modules)
– Lakehouse Fundamentals (bronze/silver/gold zone demos)
– ELT jobs (SQL, Python transformations writing to Delta)
– Streaming/Incremental (Structured Streaming jobs, Auto Loader ingestion)
– Production Pipelines (Delta Live Tables definitions, orchestration tasks)
– Governance (Unity Catalog or ACLs on tables)
e. Supporting Scripts & Assets
– “Includes” for environment setup and shared helper code
– “Labs” for hands-on exercises, with “Solutions” side-by-side
Relationships and data flows
• Flow from Auto Loader bronze tables → transformations in ELT scripts (module 2) → silver/gold Delta tables.
• Delta Live Tables pipelines pulling from silver/gold tables and writing curated outputs.
Architectural patterns
• Medallion (bronze, silver, gold) layered data architecture
• Declarative pipeline (Delta Live Tables) vs. imperative notebooks
• Streaming ingestion → micro-batches (Structured Streaming + Auto Loader)
• Separation of concerns: raw ingestion, ETL/ELT, serving layer, governance
Final advice
• Be as granular as practical: break out clusters vs. pipelines, notebooks vs. SQL jobs, raw vs. curated zones.
• Resist over-crowding: group related notebooks/scripts into single “Lakehouse Fundamentals” or “Incremental Processing” blocks when needed.