How immanuel1987/databricks_sample_project works
This is a Databricks aviation-data engineering project whose main workflow ingests source files incrementally, transforms them through Bronze and Silver, then builds Gold dimensions and a bookings fact table for analytics. Job parameters dispatch per-source Bronze ingestion; Lakeflow-style transformations define the Silver tables. The Gold notebooks implement incremental CDC processing and dimensional modeling. The README also describes a stakeholder KPI dashboard, but no dashboard implementation appears in the supplied tree. Edges reflect the excerpts and documented workflow; unsampled pipeline internals are not inferred.
Main entry points: bronze_ingestion_job.py Ingestion job · 02_Bronze_Layer.py Auto Loader ingestion · 06_Gold_Fact.py Fact builder · 01_Setup.py Volumes and schemas
Components 16
Workspace setup
- Volumes and schemas
notebooks/01_Setup.py
Bronze ingestion
- Ingestion job
jobs/bronze_ingestion_job.py - Source parameters
notebooks/03_Src_Parameters.py - Auto Loader ingestion
notebooks/02_Bronze_Layer.py - Bronze Delta data
Silver transformations
- Bookings Silver pipeline
pipelines/01_Bookings_Silver_Pipeline.py - Streaming transformations
notebooks/04_Silver_Layer.py - Silver tables
- Flights project pipeline
pipelines/02_Flights_Project_Pipeline.py
Gold warehouse
- Dimension builder
notebooks/05_Gold_Dims.py - Gold dimensions
- Fact builder
notebooks/06_Gold_Fact.py - Bookings fact
Other components
- Data engineer
- Aviation source files
data - Databricks platform
Connections 17
- Data engineer → Volumes and schemas: provisions
notebooks/01_Setup.py - Volumes and schemas → Databricks platform: creates volumes
notebooks/01_Setup.py - Data engineer → Ingestion job: runs job
- Ingestion job → Source parameters: runs first
jobs/bronze_ingestion_job.py - Ingestion job → Auto Loader ingestion: dispatches sources
jobs/bronze_ingestion_job.py - Source parameters → Ingestion job: sets task values
notebooks/03_Src_Parameters.py - Aviation source files → Auto Loader ingestion: loads files
README.md - Auto Loader ingestion → Bronze Delta data: writes Delta
README.md - Bronze Delta data → Streaming transformations: streams Delta
notebooks/04_Silver_Layer.py - Streaming transformations → Silver tables: defines tables
notebooks/04_Silver_Layer.py - Bookings Silver pipeline → Silver tables: orchestrates
- Flights project pipeline → Silver tables: orchestrates
- Silver tables → Dimension builder: reads sources
notebooks/05_Gold_Dims.py - Dimension builder → Gold dimensions: upserts dimensions
notebooks/05_Gold_Dims.py - Silver tables → Fact builder: reads bookings
notebooks/06_Gold_Fact.py - Gold dimensions → Fact builder: joins dimensions
notebooks/06_Gold_Fact.py - Fact builder → Bookings fact: upserts fact
notebooks/06_Gold_Fact.py