# tangybluff/de-healthcare-pipeline architecture

> Architecture diagram of the GitHub repository tangybluff/de-healthcare-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tangybluff/de-healthcare-pipeline
- Repository: https://github.com/tangybluff/de-healthcare-pipeline

## Overview

##### Project Type and Purpose
- This is an end-to-end batch data pipeline (full-stack data engineering project) for COVID-19 analytics.
- It ingests raw data from Kaggle into GCP storage, transforms it through a medallion (staging/intermediate/marts) architecture via dbt, orchestrates workflow with Dagster, manages infra via Terraform, and visualizes results in Looker.

##### Main Components
a. Infrastructure Layer (Terraform)
- GCP Resources: Cloud Storage bucket (bronze), BigQuery datasets (staging, intermediate, marts), service accounts
b. Ingestion Layer (DLT)
- Python scripts/notebooks in dlt-gcp-ingestion: Kaggle API → GCS → BigQuery (bronze)
c. Orchestration Layer (Dagster)
- dagster_project assets & schedules: pipelines that trigger ingestion, transformations, and tests
d. Transformation Layer (dbt)
- dbt models arranged in staging, intermediate, marts folders executing SQL transformations in BigQuery
e. Visualization Layer (Looker)
- Looker Studio dashboards and static PDF/PNG assets reading from marts
f. Assets & Documentation

##### Architectural Patterns and Principles
- Medallion Architecture: bronze (staging), silver (intermediate), gold (marts)
- ELT Pattern: Extract and Load raw, then Transform in-place in BigQuery
- IaC and Reproducibility: Terraform config under version control
- Modular Orchestration: Dagster assets & schedules define reusable pipeline steps
- Clear Separation of Concerns: ingestion, orchestration, transformation, visualization

##### Detailed Element Checklist
- Looker: dashboards connecting to BigQuery marts
- External: Kaggle API, GCP APIs (IAM, BigQuery, Storage)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    KaggleAPI["Kaggle API"]:::external
    GCPAPIs["GCP APIs (IAM, BigQuery, Storage)"]:::external

    %% Infrastructure Layer
    subgraph "Infrastructure Layer" 
        direction TB
        TerraformMain["main.tf"]:::infra
        TerraformVars["variables.tf"]:::infra
        ServiceAccounts["Service Accounts"]:::infra
        GCSBucket["GCS Bucket (bronze)"]:::storage
        BQDatasets["BigQuery Datasets (staging, intermediate, marts)"]:::storage
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer (DLT)" 
        direction TB
        PyScript["kaggle_dlt_extract_load.py"]:::ingest
        Notebook["kaggle-dlt-extract-load.ipynb"]:::ingest
        ManualPipeline["manual_pipeline_v1.py"]:::ingest
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer (Dagster)" 
        direction TB
        AssetsPy["assets.py"]:::orchestration
        SchedulesPy["schedules.py"]:::orchestration
        Workspace["workspace.yaml"]:::orchestration
        SetupPy["setup.py"]:::orchestration
        Dagster["Dagster Orchestrator"]:::orchestration
    end

    %% Transformation Layer
    subgraph "Transformation Layer (dbt)" 
        direction TB
        DBTProject["dbt_project.yml"]:::transform
        StagingModels["models/staging"]:::transform
        IntermediateModels["models/intermediate"]:::transform
        MartsModels["models/marts"]:::transform
        Macros["macros"]:::transform
        Analyses["analyses"]:::transform
    end

    %% Visualization Layer
    subgraph "Visualization Layer (Looker)" 
        direction TB
        LookerDash["Looker Studio Dashboards"]:::visualization
        PDFReport["COVIDEpidemetrics.pdf"]:::visualization
        LookerReadme["README.md"]:::visualization
    end

    %% Data Flow and Dependencies
    GCPAPIs -->|provisions| TerraformMain
    GCPAPIs -->|provisions| TerraformVars
    TerraformMain -->|creates| GCSBucket
    TerraformMain -->|creates| BQDatasets
    TerraformVars -->|configures| ServiceAccounts
    KaggleAPI -->|extracts raw CSV| PyScript
    KaggleAPI -->|extracts raw CSV| Notebook
    Dagster -->|triggers ingestion| PyScript
    Dagster -->|triggers ingestion| ManualPipeline
    PyScript -->|loads CSV| GCSBucket
    GCSBucket -->|loads to| BQDatasets
    Dagster -->|runs dbt| DBTProject
    DBTProject -->|runs staging models| StagingModels
    StagingModels -->|transforms to intermediate| IntermediateModels
    IntermediateModels -->|builds marts| MartsModels
    MartsModels -->|serves data| LookerDash
    LookerDash -->|exports| PDFReport

    %% Orchestration Components into Dagster
    AssetsPy --> Dagster
    SchedulesPy --> Dagster
    Workspace --> Dagster
    SetupPy --> Dagster

    %% Click Events
    click TerraformMain "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/terraform-iac/main.tf"
    click TerraformVars "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/terraform-iac/variables.tf"
    click PyScript "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dlt-gcp-ingestion/kaggle_dlt_extract_load.py"
    click Notebook "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dlt-gcp-ingestion/kaggle-dlt-extract-load.ipynb"
    click ManualPipeline "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dlt-gcp-ingestion/manual_pipeline_v1.py"
    click AssetsPy "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dagster-orchestration/dagster_project/assets.py"
    click SchedulesPy "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dagster-orchestration/dagster_project/schedules.py"
    click Workspace "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dagster-orchestration/workspace.yaml"
    click SetupPy "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dagster-orchestration/setup.py"
    click DBTProject "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/dbt-transformations/dbt_project.yml"
    click StagingModels "https://github.com/tangybluff/de-healthcare-pipeline/tree/main/dbt-transformations/models/staging"
    click IntermediateModels "https://github.com/tangybluff/de-healthcare-pipeline/tree/main/dbt-transformations/models/intermediate"
    click MartsModels "https://github.com/tangybluff/de-healthcare-pipeline/tree/main/dbt-transformations/models/marts"
    click Macros "https://github.com/tangybluff/de-healthcare-pipeline/tree/main/dbt-transformations/macros"
    click Analyses "https://github.com/tangybluff/de-healthcare-pipeline/tree/main/dbt-transformations/analyses"
    click PDFReport "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/looker-dashboard/COVIDEpidemetrics.pdf"
    click LookerReadme "https://github.com/tangybluff/de-healthcare-pipeline/blob/main/looker-dashboard/README.md"

    %% Styles
    classDef infra fill:#DDDDDD,stroke:#888,stroke-width:1px
    classDef storage fill:#CCCCCC,stroke:#444,stroke-width:1px,stroke-dasharray: 5 5
    classDef ingest fill:#AED6F1,stroke:#2E86C1,stroke-width:1px
    classDef orchestration fill:#FAD7A0,stroke:#D68910,stroke-width:1px
    classDef transform fill:#A9DFBF,stroke:#27AE60,stroke-width:1px
    classDef visualization fill:#D7BDE2,stroke:#8E44AD,stroke-width:1px
    classDef external fill:#F5CBA7,stroke:#D35400,stroke-width:1px,stroke-dasharray: 2 2
```
