# rchdev1998-jpg/etl_pipeline_databricks architecture

> Architecture diagram of the GitHub repository rchdev1998-jpg/etl_pipeline_databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rchdev1998-jpg/etl_pipeline_databricks
- Repository: https://github.com/rchdev1998-jpg/etl_pipeline_databricks

## Overview

This is a PySpark/Databricks lakehouse ETL project implementing a CRM/ERP medallion pipeline. Source CSVs are ingested into Bronze, Silver applies configured cleaning and SCD Type 2 or overwrite loads, and Gold builds customer/product dimensions and a sales fact for analytics. Logging is a shared operational layer. The README documents broader data-quality and analytical capabilities; the graph reflects sampled implementations and documented consumption. Relationships not visible in the sampled code are omitted.

Main entry points: [`code/bronze.py`](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze.py) (Bronze ingestion), [`code/silver.py`](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver.py) (Silver transforms), [`code/log.py`](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/log.py) (ETL logging), [`code/gold.py`](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold.py) (Gold publishing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Source ingestion"]
  node_bronze_process["Bronze ingestion<br/>[bronze.py]"]
  node_bronze_config["Ingestion settings<br/>[bronze_config.py]"]
  node_bronze_tables[("Bronze tables<br/>[bronze.py]")]
end

subgraph group_refinement["Data refinement"]
  node_silver_process["Silver transforms<br/>[silver.py]"]
  node_silver_config["Transform settings<br/>[silver_config.py]"]
  node_silver_tables[("Silver tables<br/>[silver.py]")]
end

subgraph group_analytics["Business analytics"]
  node_gold_process["Gold publishing<br/>[gold.py]"]
  node_gold_config["Business models<br/>[gold_config.py]"]
  node_gold_tables[("Gold datasets<br/>[gold.py]")]
end

subgraph group_operations["Pipeline operations"]
  node_logs["ETL logging<br/>[log.py]"]
  node_log_store[("ETL log tables<br/>[log.py]")]
end

node_crm["CRM files"]
node_erp["ERP files"]
node_analysts(("Analysts / BI"))

node_crm -->|"provides CSVs"| node_bronze_process
node_erp -->|"provides CSVs"| node_bronze_process
node_bronze_process -->|"loads settings"| node_bronze_config
node_bronze_process -->|"writes tables"| node_bronze_tables
node_bronze_process -->|"records runs"| node_logs
node_silver_process -->|"loads settings"| node_silver_config
node_silver_process -->|"queries sources"| node_bronze_tables
node_silver_process -->|"loads tables"| node_silver_tables
node_silver_process -->|"records runs"| node_logs
node_gold_process -->|"loads settings"| node_gold_config
node_gold_config -->|"queries and joins"| node_silver_tables
node_gold_process -->|"publishes tables"| node_gold_tables
node_gold_process -->|"records runs"| node_logs
node_logs -->|"inserts summaries"| node_log_store
node_gold_tables -->|"serves analytics"| node_analysts

click node_bronze_process "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze.py"
click node_bronze_config "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze_config.py"
click node_bronze_tables "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze.py"
click node_silver_process "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver.py"
click node_silver_config "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver_config.py"
click node_silver_tables "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver.py"
click node_gold_process "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold.py"
click node_gold_config "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold_config.py"
click node_gold_tables "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold.py"
click node_logs "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/log.py"
click node_log_store "https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/log.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bronze_process,node_bronze_config,node_bronze_tables toneBlue
class node_silver_process,node_silver_config,node_silver_tables toneAmber
class node_gold_process,node_gold_config,node_gold_tables toneMint
class node_logs,node_log_store toneRose
class node_crm,node_erp,node_analysts toneIndigo
```

## Components

### Source ingestion

- [Bronze ingestion](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze.py): `code/bronze.py`
- [Ingestion settings](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze_config.py): `code/bronze_config.py`
- [Bronze tables](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/bronze.py): `code/bronze.py`

### Data refinement

- [Silver transforms](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver.py): `code/silver.py`
- [Transform settings](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver_config.py): `code/silver_config.py`
- [Silver tables](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/silver.py): `code/silver.py`

### Business analytics

- [Gold publishing](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold.py): `code/gold.py`
- [Business models](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold_config.py): `code/gold_config.py`
- [Gold datasets](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/gold.py): `code/gold.py`

### Pipeline operations

- [ETL logging](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/log.py): `code/log.py`
- [ETL log tables](https://github.com/rchdev1998-jpg/etl_pipeline_databricks/blob/main/code/log.py): `code/log.py`

### Other components

- **CRM files**
- **ERP files**
- **Analysts / BI**

## Connections

- CRM files → Bronze ingestion: provides CSVs (evidence: `code/bronze.py`)
- ERP files → Bronze ingestion: provides CSVs (evidence: `code/bronze.py`)
- Bronze ingestion → Ingestion settings: loads settings (evidence: `code/bronze.py`)
- Bronze ingestion → Bronze tables: writes tables (evidence: `code/bronze.py`)
- Bronze ingestion → ETL logging: records runs (evidence: `code/bronze.py`)
- Silver transforms → Transform settings: loads settings (evidence: `code/silver.py`)
- Silver transforms → Bronze tables: queries sources
- Silver transforms → Silver tables: loads tables (evidence: `code/silver.py`)
- Silver transforms → ETL logging: records runs (evidence: `code/silver.py`)
- Gold publishing → Business models: loads settings (evidence: `code/gold.py`)
- Business models → Silver tables: queries and joins (evidence: `code/gold_config.py`)
- Gold publishing → Gold datasets: publishes tables (evidence: `code/gold.py`)
- Gold publishing → ETL logging: records runs (evidence: `code/gold.py`)
- ETL logging → ETL log tables: inserts summaries (evidence: `code/log.py`)
- Gold datasets → Analysts / BI: serves analytics (evidence: `README.md`)