# wilfex81/data-warehouse-project architecture

> Architecture diagram of the GitHub repository wilfex81/data-warehouse-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wilfex81/data-warehouse-project
- Repository: https://github.com/wilfex81/data-warehouse-project

## Overview

This repository implements a CRM/ERP lakehouse warehouse: source CSVs land in S3, Databricks Auto Loader creates bronze Delta tables, dbt builds silver staging models and gold star-schema outputs, and Airflow triggers the Databricks job. Terraform provisions the documented AWS and Unity Catalog infrastructure. The README documents the job-task chain and major capabilities; the sampled source confirms the ingestion behavior and Airflow trigger, but not the internals of the dbt models, job definition, or Terraform resources.

Main entry points: [`bundle/resources/jobs.yml`](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml) (dbt run task), [`dbt/models/silver`](https://github.com/wilfex81/data-warehouse-project/tree/main/dbt/models/silver) (Silver staging), [`infra/environments/dev/main.tf`](https://github.com/wilfex81/data-warehouse-project/blob/main/infra/environments/dev/main.tf) (Dev infrastructure), [`bundle/src/ingestion/autoloader_bronze.py`](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/src/ingestion/autoloader_bronze.py) (Auto Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow_dag["Airflow DAG"]
  node_databricks_jobs["Databricks job<br/>[jobs.yml]"]
  node_bronze_task["Bronze task<br/>[jobs.yml]"]
  node_dbt_run_task["dbt run task<br/>[jobs.yml]"]
  node_dbt_test_task["dbt test task<br/>[jobs.yml]"]
end

subgraph group_ingestion["Ingestion"]
  node_autoloader["Auto Loader"]
  node_bronze[("Bronze Delta<br/>[sources.yml]")]
end

subgraph group_transform["Transformation"]
  node_silver_models["Silver staging"]
  node_gold_customers["Customer dimension<br/>[dim_customers.sql]"]
  node_gold_products["Product dimension<br/>[dim_products.sql]"]
  node_gold_sales["Sales fact<br/>[fct_sales.sql]"]
end

subgraph group_platform["Platform"]
  node_s3_lake[("AWS S3 lake<br/>[main.tf]")]
  node_unity_catalog[("Unity Catalog<br/>[main.tf]")]
  node_iam["Databricks IAM<br/>[main.tf]"]
  node_terraform_dev["Dev infrastructure<br/>[main.tf]"]
end

node_airflow_actor(("Airflow operator"))
node_source_csv["CRM / ERP CSVs"]

node_airflow_actor -->|"triggers"| node_airflow_dag
node_airflow_dag -->|"runs job"| node_databricks_jobs
node_databricks_jobs -->|"dispatches"| node_bronze_task
node_bronze_task -->|"runs"| node_autoloader
node_databricks_jobs -->|"then dispatches"| node_dbt_run_task
node_dbt_run_task -->|"builds"| node_silver_models
node_dbt_run_task -->|"builds"| node_gold_customers
node_dbt_run_task -->|"builds"| node_gold_products
node_dbt_run_task -->|"builds"| node_gold_sales
node_databricks_jobs -->|"then dispatches"| node_dbt_test_task
node_source_csv -->|"lands in"| node_s3_lake
node_autoloader -->|"reads raw files"| node_s3_lake
node_autoloader -->|"writes Delta"| node_bronze
node_silver_models -->|"reads sources"| node_bronze
node_gold_customers -->|"reads staging"| node_silver_models
node_gold_products -->|"reads staging"| node_silver_models
node_gold_sales -->|"reads staging"| node_silver_models
node_terraform_dev -->|"provisions"| node_s3_lake
node_terraform_dev -->|"provisions"| node_unity_catalog
node_terraform_dev -->|"provisions"| node_iam
node_unity_catalog -->|"governs"| node_bronze

click node_airflow_dag "https://github.com/wilfex81/data-warehouse-project/blob/main/orchestration/airflow/dags/dw_medallion_pipeline.py"
click node_databricks_jobs "https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml"
click node_bronze_task "https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml"
click node_dbt_run_task "https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml"
click node_dbt_test_task "https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml"
click node_source_csv "https://github.com/wilfex81/data-warehouse-project/tree/main/datasets"
click node_s3_lake "https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/aws_s3_lake/main.tf"
click node_autoloader "https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/src/ingestion/autoloader_bronze.py"
click node_bronze "https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/bronze/sources.yml"
click node_silver_models "https://github.com/wilfex81/data-warehouse-project/tree/main/dbt/models/silver"
click node_gold_customers "https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/dim_customers.sql"
click node_gold_products "https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/dim_products.sql"
click node_gold_sales "https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/fct_sales.sql"
click node_unity_catalog "https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/unity_catalog/main.tf"
click node_iam "https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/aws_iam_databricks/main.tf"
click node_terraform_dev "https://github.com/wilfex81/data-warehouse-project/blob/main/infra/environments/dev/main.tf"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow_dag,node_databricks_jobs,node_bronze_task,node_dbt_run_task,node_dbt_test_task toneBlue
class node_autoloader,node_bronze toneAmber
class node_silver_models,node_gold_customers,node_gold_products,node_gold_sales toneMint
class node_s3_lake,node_unity_catalog,node_iam,node_terraform_dev toneRose
class node_airflow_actor toneIndigo
class node_source_csv toneTeal
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/wilfex81/data-warehouse-project/blob/main/orchestration/airflow/dags/dw_medallion_pipeline.py): `orchestration/airflow/dags/dw_medallion_pipeline.py`
- [Databricks job](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml): `bundle/resources/jobs.yml`
- [Bronze task](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml): `bundle/resources/jobs.yml`
- [dbt run task](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml): `bundle/resources/jobs.yml`
- [dbt test task](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/resources/jobs.yml): `bundle/resources/jobs.yml`

### Ingestion

- [Auto Loader](https://github.com/wilfex81/data-warehouse-project/blob/main/bundle/src/ingestion/autoloader_bronze.py): `bundle/src/ingestion/autoloader_bronze.py`
- [Bronze Delta](https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/bronze/sources.yml): `dbt/models/bronze/sources.yml`

### Transformation

- [Silver staging](https://github.com/wilfex81/data-warehouse-project/tree/main/dbt/models/silver): `dbt/models/silver`
- [Customer dimension](https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/dim_customers.sql): `dbt/models/gold/dim_customers.sql`
- [Product dimension](https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/dim_products.sql): `dbt/models/gold/dim_products.sql`
- [Sales fact](https://github.com/wilfex81/data-warehouse-project/blob/main/dbt/models/gold/fct_sales.sql): `dbt/models/gold/fct_sales.sql`

### Platform

- [AWS S3 lake](https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/aws_s3_lake/main.tf): `infra/modules/aws_s3_lake/main.tf`
- [Unity Catalog](https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/unity_catalog/main.tf): `infra/modules/unity_catalog/main.tf`
- [Databricks IAM](https://github.com/wilfex81/data-warehouse-project/blob/main/infra/modules/aws_iam_databricks/main.tf): `infra/modules/aws_iam_databricks/main.tf`
- [Dev infrastructure](https://github.com/wilfex81/data-warehouse-project/blob/main/infra/environments/dev/main.tf): `infra/environments/dev/main.tf`

### Other components

- **Airflow operator**
- [CRM / ERP CSVs](https://github.com/wilfex81/data-warehouse-project/tree/main/datasets): `datasets`

## Connections

- Airflow operator → Airflow DAG: triggers
- Airflow DAG → Databricks job: runs job
- Databricks job → Bronze task: dispatches
- Bronze task → Auto Loader: runs
- Databricks job → dbt run task: then dispatches
- dbt run task → Silver staging: builds
- dbt run task → Customer dimension: builds
- dbt run task → Product dimension: builds
- dbt run task → Sales fact: builds
- Databricks job → dbt test task: then dispatches
- CRM / ERP CSVs → AWS S3 lake: lands in
- Auto Loader → AWS S3 lake: reads raw files
- Auto Loader → Bronze Delta: writes Delta
- Silver staging → Bronze Delta: reads sources
- Customer dimension → Silver staging: reads staging
- Product dimension → Silver staging: reads staging
- Sales fact → Silver staging: reads staging
- Dev infrastructure → AWS S3 lake: provisions
- Dev infrastructure → Unity Catalog: provisions
- Dev infrastructure → Databricks IAM: provisions
- Unity Catalog → Bronze Delta: governs