# bigdatabysumitm/trendytech-azure-project architecture

> Architecture diagram of the GitHub repository bigdatabysumitm/trendytech-azure-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigdatabysumitm/trendytech-azure-project
- Repository: https://github.com/bigdatabysumitm/trendytech-azure-project

## Overview

This repository implements a Databricks healthcare-data lakehouse: hospital, claims, and code data are transformed into Silver records and Gold analytical tables. The clearest sampled flows read hospital Parquet from Bronze, apply quality checks and history handling in Silver, then build selected Gold dimensions. The tree also documents API extracts and additional Silver/Gold domains, which are represented without assuming unsampled transformations or orchestration. Source-to-Bronze ingestion and API request details are not established by the sampled code.

Main entry points: [`3. Silver/Claims.py`](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Claims.py) (Claims), [`3. Silver/CPT codes.py`](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/CPT%20codes.py) (CPT codes), [`3. Silver/Patient.py`](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Patient.py) (Patients), [`3. Silver/Transactions.py`](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Transactions.py) (Transactions)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Lakehouse storage"]
  node_bronze[("Bronze lake")]
  node_landing[("Landing lake")]
  node_silver_store[("Silver Delta")]
  node_gold_store[("Gold tables")]
end

subgraph group_silver["Silver transformations"]
  node_claims["Claims<br/>[Claims.py]"]
  node_cpt["CPT codes<br/>[CPT codes.py]"]
  node_patients["Patients<br/>[Patient.py]"]
  node_transactions["Transactions<br/>[Transactions.py]"]
  node_encounters["Encounters<br/>[Encounters.py]"]
  node_departments["Departments<br/>[Departments_F.py]"]
  node_providers["Providers<br/>[Providers_F.py]"]
  node_icd_silver["ICD codes<br/>[ICD Code.ipynb]"]
  node_npi_silver["NPI data<br/>[NPI.ipynb]"]
end

subgraph group_gold["Gold analytics"]
  node_gold_patient["Patient dimension<br/>[dim_patient.py]"]
  node_gold_cpt["CPT dimension<br/>[dim_cpt_code.py]"]
  node_gold_department["Department dimension<br/>[dim_department.py]"]
  node_gold_provider["Provider dimension<br/>[dim_provider.py]"]
  node_gold_icd["ICD dimension<br/>[dim_icd_code.ipynb]"]
  node_gold_npi["NPI dimension<br/>[dim_npi.ipynb]"]
  node_gold_fact["Transaction fact"]
end

subgraph group_reference["Reference data"]
  node_icd_extract["ICD API extract"]
  node_npi_extract["NPI API extract"]
end

node_hospitals["Hospital datasets"]
node_claims_files["Claims files"]
node_cpt_files["CPT file"]
node_api_services(("Code APIs"))

node_claims_files -->|"stored as"| node_landing
node_cpt_files -->|"stored as"| node_landing
node_landing -->|"CSV read"| node_claims
node_claims -->|"writes Parquet"| node_bronze
node_landing -->|"CSV read"| node_cpt
node_cpt -->|"writes Parquet"| node_bronze
node_bronze -->|"Parquet read"| node_patients
node_bronze -->|"Parquet read"| node_transactions
node_bronze -->|"Parquet read"| node_encounters
node_bronze -->|"Parquet read"| node_departments
node_bronze -->|"Parquet read"| node_providers
node_claims -->|"merges records"| node_silver_store
node_cpt -->|"merges records"| node_silver_store
node_patients -->|"merges records"| node_silver_store
node_transactions -->|"merges records"| node_silver_store
node_encounters -->|"merges records"| node_silver_store
node_departments -->|"writes table"| node_silver_store
node_providers -->|"writes table"| node_silver_store
node_silver_store -->|"reads current"| node_gold_patient
node_silver_store -->|"reads current"| node_gold_cpt
node_silver_store -->|"reads"| node_gold_department
node_silver_store -->|"reads"| node_gold_provider
node_silver_store -->|"reads"| node_gold_fact
node_gold_patient -->|"writes dimension"| node_gold_store
node_gold_cpt -->|"writes dimension"| node_gold_store
node_gold_department -->|"writes dimension"| node_gold_store
node_gold_provider -->|"writes dimension"| node_gold_store
node_gold_fact -->|"writes fact"| node_gold_store
node_api_services -.->|"API source"| node_icd_extract
node_api_services -.->|"API source"| node_npi_extract
node_icd_extract -.->|"feeds codes"| node_icd_silver
node_npi_extract -.->|"feeds data"| node_npi_silver
node_icd_silver -.->|"stores codes"| node_silver_store
node_npi_silver -.->|"stores data"| node_silver_store
node_silver_store -.->|"reads"| node_gold_icd
node_silver_store -.->|"reads"| node_gold_npi
node_gold_icd -.->|"writes dimension"| node_gold_store
node_gold_npi -.->|"writes dimension"| node_gold_store

click node_hospitals "https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/EMR"
click node_claims_files "https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/claims"
click node_cpt_files "https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/cptcodes"
click node_claims "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Claims.py"
click node_cpt "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/CPT%20codes.py"
click node_patients "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Patient.py"
click node_transactions "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Transactions.py"
click node_encounters "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Encounters.py"
click node_departments "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Departments_F.py"
click node_providers "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Providers_F.py"
click node_icd_extract "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/2.%20API%20extracts/ICD%20Code%20API%20extract.ipynb"
click node_npi_extract "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/2.%20API%20extracts/NPI%20API%20extract.ipynb"
click node_icd_silver "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/ICD%20Code.ipynb"
click node_npi_silver "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/NPI.ipynb"
click node_gold_patient "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_patient.py"
click node_gold_cpt "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_cpt_code.py"
click node_gold_department "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_department.py"
click node_gold_provider "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_provider.py"
click node_gold_icd "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_icd_code.ipynb"
click node_gold_npi "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_npi.ipynb"
click node_gold_fact "https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/fact_transaction.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bronze,node_landing,node_silver_store,node_gold_store toneBlue
class node_claims,node_cpt,node_patients,node_transactions,node_encounters,node_departments,node_providers,node_icd_silver,node_npi_silver toneAmber
class node_gold_patient,node_gold_cpt,node_gold_department,node_gold_provider,node_gold_icd,node_gold_npi,node_gold_fact,node_api_services toneMint
class node_icd_extract,node_npi_extract toneRose
class node_hospitals,node_claims_files,node_cpt_files toneTeal
```

## Components

### Lakehouse storage

- **Bronze lake**
- **Landing lake**
- **Silver Delta**
- **Gold tables**

### Silver transformations

- [Claims](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Claims.py): `3. Silver/Claims.py`
- [CPT codes](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/CPT%20codes.py): `3. Silver/CPT codes.py`
- [Patients](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Patient.py): `3. Silver/Patient.py`
- [Transactions](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Transactions.py): `3. Silver/Transactions.py`
- [Encounters](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Encounters.py): `3. Silver/Encounters.py`
- [Departments](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Departments_F.py): `3. Silver/Departments_F.py`
- [Providers](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/Providers_F.py): `3. Silver/Providers_F.py`
- [ICD codes](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/ICD%20Code.ipynb): `3. Silver/ICD Code.ipynb`
- [NPI data](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/3.%20Silver/NPI.ipynb): `3. Silver/NPI.ipynb`

### Gold analytics

- [Patient dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_patient.py): `4. Gold/dim_patient.py`
- [CPT dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_cpt_code.py): `4. Gold/dim_cpt_code.py`
- [Department dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_department.py): `4. Gold/dim_department.py`
- [Provider dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_provider.py): `4. Gold/dim_provider.py`
- [ICD dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_icd_code.ipynb): `4. Gold/dim_icd_code.ipynb`
- [NPI dimension](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/dim_npi.ipynb): `4. Gold/dim_npi.ipynb`
- [Transaction fact](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/4.%20Gold/fact_transaction.sql): `4. Gold/fact_transaction.sql`

### Reference data

- [ICD API extract](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/2.%20API%20extracts/ICD%20Code%20API%20extract.ipynb): `2. API extracts/ICD Code API extract.ipynb`
- [NPI API extract](https://github.com/bigdatabysumitm/trendytech-azure-project/blob/main/2.%20API%20extracts/NPI%20API%20extract.ipynb): `2. API extracts/NPI API extract.ipynb`

### Other components

- [Hospital datasets](https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/EMR): `datasets/EMR`
- [Claims files](https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/claims): `datasets/claims`
- [CPT file](https://github.com/bigdatabysumitm/trendytech-azure-project/tree/main/datasets/cptcodes): `datasets/cptcodes`
- **Code APIs**

## Connections

- Claims files → Landing lake: stored as
- CPT file → Landing lake: stored as
- Landing lake → Claims: CSV read
- Claims → Bronze lake: writes Parquet
- Landing lake → CPT codes: CSV read
- CPT codes → Bronze lake: writes Parquet
- Bronze lake → Patients: Parquet read
- Bronze lake → Transactions: Parquet read
- Bronze lake → Encounters: Parquet read
- Bronze lake → Departments: Parquet read
- Bronze lake → Providers: Parquet read
- Claims → Silver Delta: merges records
- CPT codes → Silver Delta: merges records
- Patients → Silver Delta: merges records
- Transactions → Silver Delta: merges records
- Encounters → Silver Delta: merges records
- Departments → Silver Delta: writes table
- Providers → Silver Delta: writes table
- Silver Delta → Patient dimension: reads current
- Silver Delta → CPT dimension: reads current
- Silver Delta → Department dimension: reads
- Silver Delta → Provider dimension: reads
- Silver Delta → Transaction fact: reads
- Patient dimension → Gold tables: writes dimension
- CPT dimension → Gold tables: writes dimension
- Department dimension → Gold tables: writes dimension
- Provider dimension → Gold tables: writes dimension
- Transaction fact → Gold tables: writes fact
- Code APIs → ICD API extract: API source
- Code APIs → NPI API extract: API source
- ICD API extract → ICD codes: feeds codes
- NPI API extract → NPI data: feeds data
- ICD codes → Silver Delta: stores codes
- NPI data → Silver Delta: stores data
- Silver Delta → ICD dimension: reads
- Silver Delta → NPI dimension: reads
- ICD dimension → Gold tables: writes dimension
- NPI dimension → Gold tables: writes dimension