How bigdatabysumitm/trendytech-azure-project works
This repository implements a Databricks healthcare-data lakehouse: hospital, claims, and code data are transformed into Silver records and Gold analytical tables. The clearest sampled flows read hospital Parquet from Bronze, apply quality checks and history handling in Silver, then build selected Gold dimensions. The tree also documents API extracts and additional Silver/Gold domains, which are represented without assuming unsampled transformations or orchestration. Source-to-Bronze ingestion and API request details are not established by the sampled code.
Main entry points: Claims.py Claims · CPT codes.py CPT codes · Patient.py Patients · Transactions.py Transactions
Components 26
Lakehouse storage
- Bronze lake
- Landing lake
- Silver Delta
- Gold tables
Silver transformations
- Claims
3. Silver/Claims.py - CPT codes
3. Silver/CPT codes.py - Patients
3. Silver/Patient.py - Transactions
3. Silver/Transactions.py - Encounters
3. Silver/Encounters.py - Departments
3. Silver/Departments_F.py - Providers
3. Silver/Providers_F.py - ICD codes
3. Silver/ICD Code.ipynb - NPI data
3. Silver/NPI.ipynb
Gold analytics
- Patient dimension
4. Gold/dim_patient.py - CPT dimension
4. Gold/dim_cpt_code.py - Department dimension
4. Gold/dim_department.py - Provider dimension
4. Gold/dim_provider.py - ICD dimension
4. Gold/dim_icd_code.ipynb - NPI dimension
4. Gold/dim_npi.ipynb - Transaction fact
4. Gold/fact_transaction.sql
Reference data
- ICD API extract
2. API extracts/ICD Code API extract.ipynb - NPI API extract
2. API extracts/NPI API extract.ipynb
Other components
- Hospital datasets
datasets/EMR - Claims files
datasets/claims - CPT file
datasets/cptcodes - Code APIs
Connections 38
- Claims files → Landing lake: stored as
- CPT file → Landing lake: stored as
- Landing lake → Claims: CSV read
- Claims → Bronze lake: writes Parquet
- Landing lake → CPT codes: CSV read
- CPT codes → Bronze lake: writes Parquet
- Bronze lake → Patients: Parquet read
- Bronze lake → Transactions: Parquet read
- Bronze lake → Encounters: Parquet read
- Bronze lake → Departments: Parquet read
- Bronze lake → Providers: Parquet read
- Claims → Silver Delta: merges records
- CPT codes → Silver Delta: merges records
- Patients → Silver Delta: merges records
- Transactions → Silver Delta: merges records
- Encounters → Silver Delta: merges records
- Departments → Silver Delta: writes table
- Providers → Silver Delta: writes table
- Silver Delta → Patient dimension: reads current
- Silver Delta → CPT dimension: reads current
- Silver Delta → Department dimension: reads
- Silver Delta → Provider dimension: reads
- Silver Delta → Transaction fact: reads
- Patient dimension → Gold tables: writes dimension
- CPT dimension → Gold tables: writes dimension
- Department dimension → Gold tables: writes dimension
- Provider dimension → Gold tables: writes dimension
- Transaction fact → Gold tables: writes fact
- Code APIs → ICD API extract: API source
- Code APIs → NPI API extract: API source
- ICD API extract → ICD codes: feeds codes
- NPI API extract → NPI data: feeds data
- ICD codes → Silver Delta: stores codes
- NPI data → Silver Delta: stores data
- Silver Delta → ICD dimension: reads
- Silver Delta → NPI dimension: reads
- ICD dimension → Gold tables: writes dimension
- NPI dimension → Gold tables: writes dimension