How levavasseuryann/airflow-bigquery works
This repository is an Airflow 3 HR analytics platform: a scheduled regional-file ingestion publishes a raw-data asset, dbt transforms that data into marts and publishes a second asset, and an independent quality DAG checks and audits the marts. The README also documents a weekly maintenance DAG, cluster policies, and local DuckDB versus production BigQuery. DAG and ingestion/quality wiring is supported by sampled source; the transformation internals, maintenance behavior, and warehouse backends are represented at the documented subsystem level because their implementation was not fully sampled.
Main entry points: ingest_hr_sources.py Ingestion DAG · warehouse.py Warehouse adapter · transform_hr_dbt.py Transformation DAG · data_quality_hr.py Quality DAG
Components 20
Airflow orchestration
- Ingestion DAG
dags/ingest_hr_sources.py - Transformation DAG
dags/transform_hr_dbt.py - Quality DAG
dags/data_quality_hr.py - Maintenance DAG
dags/platform_maintenance.py - Data assets
src/hr_pipeline/assets.py
Regional ingestion
- Discover sources
src/hr_pipeline/ingestion.py - Validate and land
src/hr_pipeline/ingestion.py - Load raw table
src/hr_pipeline/ingestion.py
dbt transformation
- dbt models, seeds, snapshots, tests
include/dbt
Data quality
- Marts check suite
src/hr_pipeline/quality/suite.py - Quality check operator
src/hr_pipeline/operators/data_quality.py - Check and report engine
src/hr_pipeline/quality/checks.py - Audit report task
dags/data_quality_hr.py
Platform services
- Runtime settings
src/hr_pipeline/config.py - Warehouse adapter
src/hr_pipeline/warehouse.py - Cluster policies
config/airflow_local_settings.py
Other components
- Pipeline operator
- Regional HR extracts
include/data/raw - DuckDB warehouse
- BigQuery warehouse
Connections 34
- Pipeline operator → Ingestion DAG: triggers
- Regional HR extracts → Discover sources: scanned by
- Ingestion DAG → Discover sources: calls
- Ingestion DAG → Validate and land: maps tasks
- Ingestion DAG → Load raw table: calls
- Discover sources → Validate and land: provides files
- Validate and land → Load raw table: provides Parquet
- Ingestion DAG → Runtime settings: reads configuration
- Ingestion DAG → Warehouse adapter: loads through
- Load raw table → Warehouse adapter: writes raw table
- Ingestion DAG → Data assets: publishes raw asset
- Data assets → Transformation DAG: schedules on raw asset
- Transformation DAG → Warehouse adapter: verifies and reads marts
- Transformation DAG → Runtime settings: reads configuration
- Transformation DAG → dbt models, seeds, snapshots, tests: runs via Cosmos
- dbt models, seeds, snapshots, tests → DuckDB warehouse: reads raw and writes marts
- dbt models, seeds, snapshots, tests → BigQuery warehouse: reads raw and writes marts
- Transformation DAG → Data assets: publishes marts asset
- Data assets → Quality DAG: schedules on marts asset
- Quality DAG → Marts check suite: builds checks from
- Quality DAG → Quality check operator: creates check tasks
- Quality check operator → Check and report engine: runs check
- Quality check operator → Warehouse adapter: queries marts
- Quality DAG → Audit report task: aggregates results
- Audit report task → Check and report engine: uses report types
- Audit report task → Warehouse adapter: appends audit rows
- Quality DAG → Runtime settings: reads configuration
- Warehouse adapter → DuckDB warehouse: connects locally
- Warehouse adapter → BigQuery warehouse: connects in production
- Maintenance DAG → Warehouse adapter: probes and inventories
- Cluster policies → Ingestion DAG: enforces DAG policy
- Cluster policies → Transformation DAG: enforces DAG policy
- Cluster policies → Quality DAG: enforces DAG policy
- Cluster policies → Maintenance DAG: enforces DAG policy