How wilfex81/data-warehouse-project works
This repository implements a CRM/ERP lakehouse warehouse: source CSVs land in S3, Databricks Auto Loader creates bronze Delta tables, dbt builds silver staging models and gold star-schema outputs, and Airflow triggers the Databricks job. Terraform provisions the documented AWS and Unity Catalog infrastructure. The README documents the job-task chain and major capabilities; the sampled source confirms the ingestion behavior and Airflow trigger, but not the internals of the dbt models, job definition, or Terraform resources.
Main entry points: jobs.yml dbt run task · silver Silver staging · main.tf Dev infrastructure · autoloader_bronze.py Auto Loader
Components 17
Orchestration
- Airflow DAG
orchestration/airflow/dags/dw_medallion_pipeline.py - Databricks job
bundle/resources/jobs.yml - Bronze task
bundle/resources/jobs.yml - dbt run task
bundle/resources/jobs.yml - dbt test task
bundle/resources/jobs.yml
Ingestion
- Auto Loader
bundle/src/ingestion/autoloader_bronze.py - Bronze Delta
dbt/models/bronze/sources.yml
Transformation
- Silver staging
dbt/models/silver - Customer dimension
dbt/models/gold/dim_customers.sql - Product dimension
dbt/models/gold/dim_products.sql - Sales fact
dbt/models/gold/fct_sales.sql
Platform
- AWS S3 lake
infra/modules/aws_s3_lake/main.tf - Unity Catalog
infra/modules/unity_catalog/main.tf - Databricks IAM
infra/modules/aws_iam_databricks/main.tf - Dev infrastructure
infra/environments/dev/main.tf
Other components
- Airflow operator
- CRM / ERP CSVs
datasets
Connections 21
- Airflow operator → Airflow DAG: triggers
- Airflow DAG → Databricks job: runs job
- Databricks job → Bronze task: dispatches
- Bronze task → Auto Loader: runs
- Databricks job → dbt run task: then dispatches
- dbt run task → Silver staging: builds
- dbt run task → Customer dimension: builds
- dbt run task → Product dimension: builds
- dbt run task → Sales fact: builds
- Databricks job → dbt test task: then dispatches
- CRM / ERP CSVs → AWS S3 lake: lands in
- Auto Loader → AWS S3 lake: reads raw files
- Auto Loader → Bronze Delta: writes Delta
- Silver staging → Bronze Delta: reads sources
- Customer dimension → Silver staging: reads staging
- Product dimension → Silver staging: reads staging
- Sales fact → Silver staging: reads staging
- Dev infrastructure → AWS S3 lake: provisions
- Dev infrastructure → Unity Catalog: provisions
- Dev infrastructure → Databricks IAM: provisions
- Unity Catalog → Bronze Delta: governs