# immanuel1987/databricks_sample_project architecture

> Architecture diagram of the GitHub repository immanuel1987/databricks_sample_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/immanuel1987/databricks_sample_project
- Repository: https://github.com/immanuel1987/databricks_sample_project

## Overview

This is a Databricks aviation-data lakehouse project. Its main workflow parameterizes file sources, incrementally ingests them into Bronze, transforms and validates streaming data in Silver, then builds incremental Gold dimensions and a booking fact table for analytics. The graph also shows job dispatch and setup. The README documents a stakeholder KPI dashboard, but no dashboard implementation appears in the supplied tree; its node is therefore only a documented capability. Edges rely on the supplied excerpts or explicit README statements.

Main entry points: [`pipelines/02_Flights_Project_Pipeline.py`](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py) (Silver pipeline), [`jobs/bronze_ingestion_job.py`](https://github.com/immanuel1987/databricks_sample_project/blob/master/jobs/bronze_ingestion_job.py) (Ingestion job), [`notebooks/02_Bronze_Layer.py`](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/02_Bronze_Layer.py) (Auto Loader ingestion), [`notebooks/06_Gold_Fact.py`](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/06_Gold_Fact.py) (Fact builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion and setup"]
  node_setup["Lakehouse setup<br/>[01_Setup.py]"]
  node_raw_volume[("Raw volume")]
  node_source_params["Source parameters"]
  node_ingestion_job["Ingestion job"]
  node_bronze_ingest["Auto Loader ingestion<br/>[02_Bronze_Layer.py]"]
  node_bronze_data[("Bronze Delta data")]
end

subgraph group_silver["Silver transformations"]
  node_silver_pipeline["Silver pipeline"]
  node_booking_stage["Booking staging"]
  node_booking_transform["Booking transformations"]
  node_silver_bookings[("Silver bookings")]
  node_silver_entities["Silver entity CDC"]
  node_silver_business["Silver business view"]
  node_silver_data[("Silver tables")]
end

subgraph group_gold["Gold analytics"]
  node_dimension_builder["Dimension builder<br/>[05_Gold_Dims.py]"]
  node_gold_dimensions[("Gold dimensions<br/>[05_Gold_Dims.py]")]
  node_fact_builder["Fact builder<br/>[06_Gold_Fact.py]"]
  node_gold_fact[("Bookings fact<br/>[06_Gold_Fact.py]")]
end

subgraph group_analytics["Analytics"]
  node_kpi_dashboard["Flight KPI dashboard"]
end

node_data_provider(("Source files"))
node_stakeholder(("Stakeholder"))

node_data_provider -->|"provides files"| node_raw_volume
node_setup -->|"creates volumes"| node_raw_volume
node_source_params -->|"sets task values"| node_ingestion_job
node_ingestion_job -->|"runs first"| node_source_params
node_ingestion_job -->|"dispatches per source"| node_bronze_ingest
node_bronze_ingest -->|"reads files"| node_raw_volume
node_bronze_ingest -->|"writes Delta"| node_bronze_data
node_bronze_data -->|"feeds transformations"| node_silver_pipeline
node_silver_pipeline -->|"registers stage"| node_booking_stage
node_booking_stage -->|"reads stage"| node_booking_transform
node_booking_transform -->|"validates and emits"| node_silver_bookings
node_silver_pipeline -->|"defines CDC flows"| node_silver_entities
node_silver_entities -->|"writes entity tables"| node_silver_data
node_silver_pipeline -->|"joins silver streams"| node_silver_business
node_silver_bookings -->|"joins bookings"| node_silver_business
node_silver_data -->|"provides source tables"| node_dimension_builder
node_dimension_builder -->|"upserts dimensions"| node_gold_dimensions
node_silver_bookings -->|"provides source facts"| node_fact_builder
node_gold_dimensions -->|"joins dimension keys"| node_fact_builder
node_fact_builder -->|"upserts fact"| node_gold_fact
node_stakeholder -.->|"views KPIs"| node_kpi_dashboard
node_gold_fact -.->|"supports analytics"| node_kpi_dashboard

click node_setup "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/01_Setup.py"
click node_source_params "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/03_Src_Parameters.py"
click node_ingestion_job "https://github.com/immanuel1987/databricks_sample_project/blob/master/jobs/bronze_ingestion_job.py"
click node_bronze_ingest "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/02_Bronze_Layer.py"
click node_silver_pipeline "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_booking_stage "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_booking_transform "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_silver_bookings "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_silver_entities "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_silver_business "https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py"
click node_dimension_builder "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/05_Gold_Dims.py"
click node_gold_dimensions "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/05_Gold_Dims.py"
click node_fact_builder "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/06_Gold_Fact.py"
click node_gold_fact "https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/06_Gold_Fact.py"
click node_kpi_dashboard "https://github.com/immanuel1987/databricks_sample_project/tree/master/notebooks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_setup,node_raw_volume,node_source_params,node_ingestion_job,node_bronze_ingest,node_bronze_data toneBlue
class node_silver_pipeline,node_booking_stage,node_booking_transform,node_silver_bookings,node_silver_entities,node_silver_business,node_silver_data toneAmber
class node_dimension_builder,node_gold_dimensions,node_fact_builder,node_gold_fact toneMint
class node_kpi_dashboard toneRose
class node_data_provider,node_stakeholder toneIndigo
```

## Components

### Ingestion and setup

- [Lakehouse setup](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/01_Setup.py): `notebooks/01_Setup.py`
- **Raw volume**
- [Source parameters](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/03_Src_Parameters.py): `notebooks/03_Src_Parameters.py`
- [Ingestion job](https://github.com/immanuel1987/databricks_sample_project/blob/master/jobs/bronze_ingestion_job.py): `jobs/bronze_ingestion_job.py`
- [Auto Loader ingestion](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/02_Bronze_Layer.py): `notebooks/02_Bronze_Layer.py`
- **Bronze Delta data**

### Silver transformations

- [Silver pipeline](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- [Booking staging](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- [Booking transformations](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- [Silver bookings](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- [Silver entity CDC](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- [Silver business view](https://github.com/immanuel1987/databricks_sample_project/blob/master/pipelines/02_Flights_Project_Pipeline.py): `pipelines/02_Flights_Project_Pipeline.py`
- **Silver tables**

### Gold analytics

- [Dimension builder](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/05_Gold_Dims.py): `notebooks/05_Gold_Dims.py`
- [Gold dimensions](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/05_Gold_Dims.py): `notebooks/05_Gold_Dims.py`
- [Fact builder](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/06_Gold_Fact.py): `notebooks/06_Gold_Fact.py`
- [Bookings fact](https://github.com/immanuel1987/databricks_sample_project/blob/master/notebooks/06_Gold_Fact.py): `notebooks/06_Gold_Fact.py`

### Analytics

- [Flight KPI dashboard](https://github.com/immanuel1987/databricks_sample_project/tree/master/notebooks): `notebooks`

### Other components

- **Source files**
- **Stakeholder**

## Connections

- Source files → Raw volume: provides files (evidence: `README.md`)
- Lakehouse setup → Raw volume: creates volumes (evidence: `notebooks/01_Setup.py`)
- Source parameters → Ingestion job: sets task values (evidence: `jobs/bronze_ingestion_job.py`)
- Ingestion job → Source parameters: runs first (evidence: `jobs/bronze_ingestion_job.py`)
- Ingestion job → Auto Loader ingestion: dispatches per source (evidence: `jobs/bronze_ingestion_job.py`)
- Auto Loader ingestion → Raw volume: reads files (evidence: `notebooks/02_Bronze_Layer.py`)
- Auto Loader ingestion → Bronze Delta data: writes Delta (evidence: `notebooks/02_Bronze_Layer.py`)
- Bronze Delta data → Silver pipeline: feeds transformations (evidence: `README.md`)
- Silver pipeline → Booking staging: registers stage (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Booking staging → Booking transformations: reads stage (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Booking transformations → Silver bookings: validates and emits (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Silver pipeline → Silver entity CDC: defines CDC flows (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Silver entity CDC → Silver tables: writes entity tables (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Silver pipeline → Silver business view: joins silver streams (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Silver bookings → Silver business view: joins bookings (evidence: `pipelines/02_Flights_Project_Pipeline.py`)
- Silver tables → Dimension builder: provides source tables (evidence: `notebooks/05_Gold_Dims.py`)
- Dimension builder → Gold dimensions: upserts dimensions (evidence: `notebooks/05_Gold_Dims.py`)
- Silver bookings → Fact builder: provides source facts (evidence: `notebooks/06_Gold_Fact.py`)
- Gold dimensions → Fact builder: joins dimension keys (evidence: `notebooks/06_Gold_Fact.py`)
- Fact builder → Bookings fact: upserts fact (evidence: `notebooks/06_Gold_Fact.py`)
- Stakeholder → Flight KPI dashboard: views KPIs (evidence: `README.md`)
- Bookings fact → Flight KPI dashboard: supports analytics (evidence: `README.md`)
