How tangybluff/de-healthcare-pipeline works
Project Type and Purpose
• This is an end-to-end batch data pipeline (full-stack data engineering project) for COVID-19 analytics.
• It ingests raw data from Kaggle into GCP storage, transforms it through a medallion (staging/intermediate/marts) architecture via dbt, orchestrates workflow with Dagster, manages infra via Terraform, and visualizes results in Looker.
Main Components
a. Infrastructure Layer (Terraform)
• GCP Resources: Cloud Storage bucket (bronze), BigQuery datasets (staging, intermediate, marts), service accounts
b. Ingestion Layer (DLT)
• Python scripts/notebooks in dlt-gcp-ingestion: Kaggle API → GCS → BigQuery (bronze)
c. Orchestration Layer (Dagster)
• dagster_project assets & schedules: pipelines that trigger ingestion, transformations, and tests
d. Transformation Layer (dbt)
• dbt models arranged in staging, intermediate, marts folders executing SQL transformations in BigQuery
e. Visualization Layer (Looker)
• Looker Studio dashboards and static PDF/PNG assets reading from marts
f. Assets & Documentation
Architectural Patterns and Principles
• Medallion Architecture: bronze (staging), silver (intermediate), gold (marts)
• ELT Pattern: Extract and Load raw, then Transform in-place in BigQuery
• IaC and Reproducibility: Terraform config under version control
• Modular Orchestration: Dagster assets & schedules define reusable pipeline steps
• Clear Separation of Concerns: ingestion, orchestration, transformation, visualization
Detailed Element Checklist
• Looker: dashboards connecting to BigQuery marts
• External: Kaggle API, GCP APIs (IAM, BigQuery, Storage)