How tuva-health/tuva works
Main Components
a. Data Sources
• Raw clinical and claims data in your warehouse (e.g., core tables loaded externally or via dbt input_layer models)
• Terminology and reference data (seeds/ S3)
b. dbt Project Layers
1) input_layer – initial “inbound” SQL models
2) staging – cleanses & standardizes inbound tables
3) normalized_input – normalization & code‐voting logic
4) intermediate – domain‐specific transforms (claims_preprocessing, service_category, etc.)
5) final/core – core data model for patient, encounter, claim, etc.
6) data_marts – specialized marts (quality_measures, cms_hcc, pqi, ed_classification, readmissions, etc.)
c. Supporting Components
• Macros library – reusable SQL functions called by models
• Seeds (seeds/terminology, seeds/value_sets) – reference & vocabulary tables loaded first
• Snapshots – temporal snapshots of key tables (cms_hcc risk, quality_measures)
• Data Quality Models & Tests – DQI models/test suites under models/data_quality & macros/tests
• Documentation – auto‐generated dbt docs site (docs/manifest.json, etc.)
• CI/CD – GitHub Actions workflows under .github/workflows, custom lint scripts, integration & CI tests
• Helper scripts – Python scripts for schema extraction & S3 loading
Relationships & Data Flow
• Snapshots to the final/core models they reference
• Data quality models consume core and marts; tests and DQI macros apply against staging & final tables
• CI/CD orchestrates:
– On commit, GitHub Actions → run dbt build/test → update docs → deploy artifacts
Architectural Patterns & Principles
• Layered ELT pipeline (modular, each folder is a logical layer)
• Reusability via macros (DRY SQL)
• Packaging & versioning (dbt package.yml, semantic versioning)
• Environment‐agnostic configuration (profiles for BigQuery, Snowflake, Redshift, Fabric)
• CI/CD gating quality (unit tests in macros/tests, integration_tests, ci_testing pipelines)
Call Out Key Technologies & Integrations
• dbt (SQL‐based transformations)
• Warehouse platforms (BigQuery, Redshift, Snowflake, Databricks, DuckDB)
• AWS S3 (for large terminology files)
• GitHub Actions (CI/CD)
• Python (helper_scripts/schema_extract.py)
• dbt docs site (static HTML in docs/)
Detail Level
• Major folder as a component (don’t list every SQL file)
• How seeds (terminology, value_sets) feed into staging