# zura16/clinical-flow architecture

> Architecture diagram of the GitHub repository zura16/clinical-flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zura16/clinical-flow
- Repository: https://github.com/zura16/clinical-flow

## Overview

ClinicalFlow is a synthetic healthcare-data lakehouse that moves FHIR, SQL Server EHR, and claims data through bronze ingestion, silver normalization and quality checks, and gold warehouse modeling for analytics. The main user workflow is to run the pipeline stages, then query warehouse outputs and operational dashboards. The graph also shows control metadata, quality/quarantine, audit, and alerting. Edges are limited to relationships documented in the README or visible in the SOURCE INDEX; unsampled capabilities are represented without inferred implementation links.

Main entry points: [`databricks/bronze/ingest_raw_data.py`](https://github.com/zura16/clinical-flow/blob/main/databricks/bronze/ingest_raw_data.py) (Bronze Ingestion), [`databricks/silver/silver_engine.py`](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/silver_engine.py) (Silver Engine), [`databricks/gold/build_facts.py`](https://github.com/zura16/clinical-flow/blob/main/databricks/gold/build_facts.py) (Fact Builder), [`databricks/utilities/failure_simulation.py`](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/failure_simulation.py) (Failure Recovery Demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source Ingestion"]
  node_bronze_ingest["Bronze Ingestion<br/>[ingest_raw_data.py]"]
  node_bronze_store[("Bronze Delta<br/>[config.py]")]
  node_sql_access["SQL Server Access<br/>[sqlserver.py]"]
end

subgraph group_silver["Silver Processing"]
  node_fhir_stage["FHIR Normalization"]
  node_relational_stage["Relational Normalization"]
  node_silver_engine["Silver Engine<br/>[silver_engine.py]"]
  node_silver_specs["Silver Mapping Specs<br/>[specs.py]"]
  node_bronze_reader["Bronze Reader<br/>[bronze_reader.py]"]
  node_quality["Quality and Quarantine<br/>[quality_engine.py]"]
  node_silver_store[("Silver Delta<br/>[config.py]")]
end

subgraph group_warehouse["Gold Warehouse"]
  node_dimensions["Dimension Builder"]
  node_facts["Fact Builder<br/>[build_facts.py]"]
  node_gold_reader["Silver Warehouse Reader<br/>[silver_reader.py]"]
  node_keys["Warehouse Keys<br/>[keys.py]"]
  node_gold_store[("Gold Delta Warehouse<br/>[config.py]")]
end

subgraph group_ops["Control and Operations"]
  node_control[("Ingestion Control<br/>[control.py]")]
  node_audit[("Run Audit<br/>[logger.py]")]
  node_alerts["Failure Alerts<br/>[alerting.py]"]
  node_failure_demo["Failure Recovery Demo"]
end

subgraph group_analytics["Analytics"]
  node_dashboards["Analytics Dashboards"]
end

node_operator(("Pipeline Operator"))
node_fhir["FHIR Bundles"]
node_ehr[("SQL Server EHR")]
node_claims["Claims Feed"]
node_webhook(("Alert Webhook"))
node_consumer(("Analytics Consumer"))

node_operator -->|"runs ingestion"| node_bronze_ingest
node_fhir -->|"supplies bundles"| node_bronze_ingest
node_ehr -->|"supplies records"| node_bronze_ingest
node_claims -->|"supplies feed"| node_bronze_ingest
node_bronze_ingest -->|"writes raw data"| node_bronze_store
node_bronze_ingest -->|"uses metadata"| node_control
node_bronze_ingest -->|"reads EHR CDC"| node_sql_access
node_operator -->|"runs stage"| node_fhir_stage
node_operator -->|"runs stage"| node_relational_stage
node_fhir_stage -->|"delegates to"| node_silver_engine
node_relational_stage -->|"delegates to"| node_silver_engine
node_silver_engine -->|"processes specs"| node_silver_specs
node_silver_engine -->|"reads bronze"| node_bronze_reader
node_silver_engine -->|"validates batches"| node_quality
node_silver_engine -->|"merges silver"| node_silver_store
node_operator -->|"runs dimensions"| node_dimensions
node_operator -->|"runs facts"| node_facts
node_dimensions -->|"builds dimensions"| node_gold_store
node_facts -->|"builds facts"| node_gold_store
node_facts -->|"reads silver"| node_gold_reader
node_facts -->|"uses key functions"| node_keys
node_dimensions -->|"uses key functions"| node_keys
node_consumer -->|"queries analytics"| node_dashboards
node_dashboards -.->|"queries warehouse"| node_gold_store
node_bronze_ingest -->|"records attempts"| node_audit
node_silver_engine -->|"records runs"| node_audit
node_bronze_ingest -->|"raises alerts"| node_alerts
node_silver_engine -->|"raises alerts"| node_alerts
node_alerts -->|"posts optionally"| node_webhook
node_operator -->|"runs recovery demo"| node_failure_demo
node_failure_demo -->|"replays ingestion"| node_bronze_ingest
node_failure_demo -->|"replays processing"| node_silver_engine
node_failure_demo -->|"corrupts source"| node_sql_access

click node_bronze_ingest "https://github.com/zura16/clinical-flow/blob/main/databricks/bronze/ingest_raw_data.py"
click node_bronze_store "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py"
click node_control "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/control.py"
click node_sql_access "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/sqlserver.py"
click node_fhir_stage "https://github.com/zura16/clinical-flow/blob/main/databricks/silver/process_fhir_silver.py"
click node_relational_stage "https://github.com/zura16/clinical-flow/blob/main/databricks/silver/process_relational_silver.py"
click node_silver_engine "https://github.com/zura16/clinical-flow/blob/main/databricks/silver/silver_engine.py"
click node_silver_specs "https://github.com/zura16/clinical-flow/blob/main/databricks/silver/specs.py"
click node_bronze_reader "https://github.com/zura16/clinical-flow/blob/main/databricks/silver/bronze_reader.py"
click node_quality "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/quality_engine.py"
click node_silver_store "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py"
click node_dimensions "https://github.com/zura16/clinical-flow/blob/main/databricks/gold/build_dimensions.py"
click node_facts "https://github.com/zura16/clinical-flow/blob/main/databricks/gold/build_facts.py"
click node_gold_reader "https://github.com/zura16/clinical-flow/blob/main/databricks/gold/silver_reader.py"
click node_keys "https://github.com/zura16/clinical-flow/blob/main/databricks/gold/keys.py"
click node_gold_store "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py"
click node_audit "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/logger.py"
click node_alerts "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/alerting.py"
click node_failure_demo "https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/failure_simulation.py"
click node_dashboards "https://github.com/zura16/clinical-flow/tree/main/dashboards"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bronze_ingest,node_bronze_store,node_sql_access toneBlue
class node_fhir_stage,node_relational_stage,node_silver_engine,node_silver_specs,node_bronze_reader,node_quality,node_silver_store,node_ehr toneAmber
class node_dimensions,node_facts,node_gold_reader,node_keys,node_gold_store,node_webhook toneMint
class node_control,node_audit,node_alerts,node_failure_demo toneRose
class node_dashboards,node_operator,node_fhir,node_claims,node_consumer toneIndigo
```

## Components

### Source Ingestion

- [Bronze Ingestion](https://github.com/zura16/clinical-flow/blob/main/databricks/bronze/ingest_raw_data.py): `databricks/bronze/ingest_raw_data.py`
- [Bronze Delta](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py): `databricks/utilities/config.py`
- [SQL Server Access](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/sqlserver.py): `databricks/utilities/sqlserver.py`

### Silver Processing

- [FHIR Normalization](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/process_fhir_silver.py): `databricks/silver/process_fhir_silver.py`
- [Relational Normalization](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/process_relational_silver.py): `databricks/silver/process_relational_silver.py`
- [Silver Engine](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/silver_engine.py): `databricks/silver/silver_engine.py`
- [Silver Mapping Specs](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/specs.py): `databricks/silver/specs.py`
- [Bronze Reader](https://github.com/zura16/clinical-flow/blob/main/databricks/silver/bronze_reader.py): `databricks/silver/bronze_reader.py`
- [Quality and Quarantine](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/quality_engine.py): `databricks/utilities/quality_engine.py`
- [Silver Delta](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py): `databricks/utilities/config.py`

### Gold Warehouse

- [Dimension Builder](https://github.com/zura16/clinical-flow/blob/main/databricks/gold/build_dimensions.py): `databricks/gold/build_dimensions.py`
- [Fact Builder](https://github.com/zura16/clinical-flow/blob/main/databricks/gold/build_facts.py): `databricks/gold/build_facts.py`
- [Silver Warehouse Reader](https://github.com/zura16/clinical-flow/blob/main/databricks/gold/silver_reader.py): `databricks/gold/silver_reader.py`
- [Warehouse Keys](https://github.com/zura16/clinical-flow/blob/main/databricks/gold/keys.py): `databricks/gold/keys.py`
- [Gold Delta Warehouse](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/config.py): `databricks/utilities/config.py`

### Control and Operations

- [Ingestion Control](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/control.py): `databricks/utilities/control.py`
- [Run Audit](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/logger.py): `databricks/utilities/logger.py`
- [Failure Alerts](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/alerting.py): `databricks/utilities/alerting.py`
- [Failure Recovery Demo](https://github.com/zura16/clinical-flow/blob/main/databricks/utilities/failure_simulation.py): `databricks/utilities/failure_simulation.py`

### Analytics

- [Analytics Dashboards](https://github.com/zura16/clinical-flow/tree/main/dashboards): `dashboards`

### Other components

- **Pipeline Operator**
- **FHIR Bundles**
- **SQL Server EHR**
- **Claims Feed**
- **Alert Webhook**
- **Analytics Consumer**

## Connections

- Pipeline Operator → Bronze Ingestion: runs ingestion (evidence: `README.md`)
- FHIR Bundles → Bronze Ingestion: supplies bundles (evidence: `README.md`)
- SQL Server EHR → Bronze Ingestion: supplies records (evidence: `README.md`)
- Claims Feed → Bronze Ingestion: supplies feed (evidence: `README.md`)
- Bronze Ingestion → Bronze Delta: writes raw data (evidence: `README.md`)
- Bronze Ingestion → Ingestion Control: uses metadata (evidence: `databricks/bronze/ingest_raw_data.py`)
- Bronze Ingestion → SQL Server Access: reads EHR CDC (evidence: `databricks/bronze/ingest_raw_data.py`)
- Pipeline Operator → FHIR Normalization: runs stage (evidence: `README.md`)
- Pipeline Operator → Relational Normalization: runs stage (evidence: `README.md`)
- FHIR Normalization → Silver Engine: delegates to (evidence: `databricks/silver/process_fhir_silver.py`)
- Relational Normalization → Silver Engine: delegates to (evidence: `databricks/silver/process_relational_silver.py`)
- Silver Engine → Silver Mapping Specs: processes specs (evidence: `databricks/silver/silver_engine.py`)
- Silver Engine → Bronze Reader: reads bronze (evidence: `databricks/silver/bronze_reader.py`)
- Silver Engine → Quality and Quarantine: validates batches (evidence: `databricks/silver/silver_engine.py`)
- Silver Engine → Silver Delta: merges silver (evidence: `databricks/silver/silver_engine.py`)
- Pipeline Operator → Dimension Builder: runs dimensions (evidence: `README.md`)
- Pipeline Operator → Fact Builder: runs facts (evidence: `README.md`)
- Dimension Builder → Gold Delta Warehouse: builds dimensions (evidence: `README.md`)
- Fact Builder → Gold Delta Warehouse: builds facts (evidence: `README.md`)
- Fact Builder → Silver Warehouse Reader: reads silver (evidence: `databricks/gold/build_facts.py`)
- Fact Builder → Warehouse Keys: uses key functions (evidence: `databricks/gold/build_facts.py`)
- Dimension Builder → Warehouse Keys: uses key functions (evidence: `databricks/gold/build_dimensions.py`)
- Analytics Consumer → Analytics Dashboards: queries analytics (evidence: `README.md`)
- Analytics Dashboards → Gold Delta Warehouse: queries warehouse (evidence: `README.md`)
- Bronze Ingestion → Run Audit: records attempts (evidence: `README.md`)
- Silver Engine → Run Audit: records runs (evidence: `databricks/silver/silver_engine.py`)
- Bronze Ingestion → Failure Alerts: raises alerts (evidence: `databricks/bronze/ingest_raw_data.py`)
- Silver Engine → Failure Alerts: raises alerts (evidence: `databricks/silver/silver_engine.py`)
- Failure Alerts → Alert Webhook: posts optionally (evidence: `databricks/utilities/alerting.py`)
- Pipeline Operator → Failure Recovery Demo: runs recovery demo (evidence: `README.md`)
- Failure Recovery Demo → Bronze Ingestion: replays ingestion (evidence: `databricks/utilities/failure_simulation.py`)
- Failure Recovery Demo → Silver Engine: replays processing (evidence: `databricks/utilities/failure_simulation.py`)
- Failure Recovery Demo → SQL Server Access: corrupts source (evidence: `databricks/utilities/failure_simulation.py`)
