# levavasseuryann/airflow-bigquery architecture

> Architecture diagram of the GitHub repository levavasseuryann/airflow-bigquery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/levavasseuryann/airflow-bigquery
- Repository: https://github.com/levavasseuryann/airflow-bigquery
- Video tour (about a minute): https://gitdiagram.com/levavasseuryann/airflow-bigquery/video

## Overview

This repository is an Airflow 3 HR analytics platform: a scheduled regional-file ingestion publishes a raw-data asset, dbt transforms that data into marts and publishes a second asset, and an independent quality DAG checks and audits the marts. The README also documents a weekly maintenance DAG, cluster policies, and local DuckDB versus production BigQuery. DAG and ingestion/quality wiring is supported by sampled source; the transformation internals, maintenance behavior, and warehouse backends are represented at the documented subsystem level because their implementation was not fully sampled.

Main entry points: [`dags/ingest_hr_sources.py`](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/ingest_hr_sources.py) (Ingestion DAG), [`src/hr_pipeline/warehouse.py`](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/warehouse.py) (Warehouse adapter), [`dags/transform_hr_dbt.py`](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/transform_hr_dbt.py) (Transformation DAG), [`dags/data_quality_hr.py`](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/data_quality_hr.py) (Quality DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Airflow orchestration"]
  node_ingest_dag["Ingestion DAG"]
  node_transform_dag["Transformation DAG"]
  node_quality_dag["Quality DAG<br/>[data_quality_hr.py]"]
  node_maintenance_dag["Maintenance DAG"]
  node_assets["Data assets<br/>[assets.py]"]
end

subgraph group_ingestion["Regional ingestion"]
  node_discover["Discover sources<br/>[ingestion.py]"]
  node_extract["Validate and land<br/>[ingestion.py]"]
  node_raw_load["Load raw table<br/>[ingestion.py]"]
end

subgraph group_transformation["dbt transformation"]
  node_dbt_project["dbt models, seeds, snapshots, tests"]
end

subgraph group_quality["Data quality"]
  node_quality_suite["Marts check suite<br/>[suite.py]"]
  node_quality_operator["Quality check operator<br/>[data_quality.py]"]
  node_quality_engine["Check and report engine<br/>[checks.py]"]
  node_audit_report["Audit report task<br/>[data_quality_hr.py]"]
end

subgraph group_platform["Platform services"]
  node_settings["Runtime settings<br/>[config.py]"]
  node_warehouse_adapter["Warehouse adapter<br/>[warehouse.py]"]
  node_cluster_policies["Cluster policies"]
end

node_operator(("Pipeline operator"))
node_regional_csv["Regional HR extracts"]
node_duckdb[("DuckDB warehouse")]
node_bigquery[("BigQuery warehouse")]

node_operator -->|"triggers"| node_ingest_dag
node_regional_csv -->|"scanned by"| node_discover
node_ingest_dag -->|"calls"| node_discover
node_ingest_dag -->|"maps tasks"| node_extract
node_ingest_dag -->|"calls"| node_raw_load
node_discover -->|"provides files"| node_extract
node_extract -->|"provides Parquet"| node_raw_load
node_ingest_dag -->|"reads configuration"| node_settings
node_ingest_dag -->|"loads through"| node_warehouse_adapter
node_raw_load -->|"writes raw table"| node_warehouse_adapter
node_ingest_dag -->|"publishes raw asset"| node_assets
node_assets -->|"schedules on raw asset"| node_transform_dag
node_transform_dag -->|"verifies and reads marts"| node_warehouse_adapter
node_transform_dag -->|"reads configuration"| node_settings
node_transform_dag -->|"runs via Cosmos"| node_dbt_project
node_dbt_project -->|"reads raw and writes marts"| node_duckdb
node_dbt_project -.->|"reads raw and writes marts"| node_bigquery
node_transform_dag -->|"publishes marts asset"| node_assets
node_assets -->|"schedules on marts asset"| node_quality_dag
node_quality_dag -->|"builds checks from"| node_quality_suite
node_quality_dag -->|"creates check tasks"| node_quality_operator
node_quality_operator -->|"runs check"| node_quality_engine
node_quality_operator -->|"queries marts"| node_warehouse_adapter
node_quality_dag -->|"aggregates results"| node_audit_report
node_audit_report -->|"uses report types"| node_quality_engine
node_audit_report -->|"appends audit rows"| node_warehouse_adapter
node_quality_dag -->|"reads configuration"| node_settings
node_warehouse_adapter -->|"connects locally"| node_duckdb
node_warehouse_adapter -.->|"connects in production"| node_bigquery
node_maintenance_dag -->|"probes and inventories"| node_warehouse_adapter
node_cluster_policies -->|"enforces DAG policy"| node_ingest_dag
node_cluster_policies -->|"enforces DAG policy"| node_transform_dag
node_cluster_policies -->|"enforces DAG policy"| node_quality_dag
node_cluster_policies -->|"enforces DAG policy"| node_maintenance_dag

click node_regional_csv "https://github.com/levavasseuryann/airflow-bigquery/tree/main/include/data/raw"
click node_ingest_dag "https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/ingest_hr_sources.py"
click node_transform_dag "https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/transform_hr_dbt.py"
click node_quality_dag "https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/data_quality_hr.py"
click node_maintenance_dag "https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/platform_maintenance.py"
click node_assets "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/assets.py"
click node_discover "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py"
click node_extract "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py"
click node_raw_load "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py"
click node_dbt_project "https://github.com/levavasseuryann/airflow-bigquery/tree/main/include/dbt"
click node_quality_suite "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/quality/suite.py"
click node_quality_operator "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/operators/data_quality.py"
click node_quality_engine "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/quality/checks.py"
click node_audit_report "https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/data_quality_hr.py"
click node_settings "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/config.py"
click node_warehouse_adapter "https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/warehouse.py"
click node_cluster_policies "https://github.com/levavasseuryann/airflow-bigquery/blob/main/config/airflow_local_settings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_dag,node_transform_dag,node_quality_dag,node_maintenance_dag,node_assets toneBlue
class node_discover,node_extract,node_raw_load,node_duckdb,node_bigquery toneAmber
class node_dbt_project toneMint
class node_quality_suite,node_quality_operator,node_quality_engine,node_audit_report toneRose
class node_settings,node_warehouse_adapter,node_cluster_policies,node_operator toneIndigo
class node_regional_csv toneTeal
```

## Components

### Airflow orchestration

- [Ingestion DAG](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/ingest_hr_sources.py): `dags/ingest_hr_sources.py`
- [Transformation DAG](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/transform_hr_dbt.py): `dags/transform_hr_dbt.py`
- [Quality DAG](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/data_quality_hr.py): `dags/data_quality_hr.py`
- [Maintenance DAG](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/platform_maintenance.py): `dags/platform_maintenance.py`
- [Data assets](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/assets.py): `src/hr_pipeline/assets.py`

### Regional ingestion

- [Discover sources](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py): `src/hr_pipeline/ingestion.py`
- [Validate and land](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py): `src/hr_pipeline/ingestion.py`
- [Load raw table](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/ingestion.py): `src/hr_pipeline/ingestion.py`

### dbt transformation

- [dbt models, seeds, snapshots, tests](https://github.com/levavasseuryann/airflow-bigquery/tree/main/include/dbt): `include/dbt`

### Data quality

- [Marts check suite](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/quality/suite.py): `src/hr_pipeline/quality/suite.py`
- [Quality check operator](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/operators/data_quality.py): `src/hr_pipeline/operators/data_quality.py`
- [Check and report engine](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/quality/checks.py): `src/hr_pipeline/quality/checks.py`
- [Audit report task](https://github.com/levavasseuryann/airflow-bigquery/blob/main/dags/data_quality_hr.py): `dags/data_quality_hr.py`

### Platform services

- [Runtime settings](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/config.py): `src/hr_pipeline/config.py`
- [Warehouse adapter](https://github.com/levavasseuryann/airflow-bigquery/blob/main/src/hr_pipeline/warehouse.py): `src/hr_pipeline/warehouse.py`
- [Cluster policies](https://github.com/levavasseuryann/airflow-bigquery/blob/main/config/airflow_local_settings.py): `config/airflow_local_settings.py`

### Other components

- **Pipeline operator**
- [Regional HR extracts](https://github.com/levavasseuryann/airflow-bigquery/tree/main/include/data/raw): `include/data/raw`
- **DuckDB warehouse**
- **BigQuery warehouse**

## Connections

- Pipeline operator → Ingestion DAG: triggers
- Regional HR extracts → Discover sources: scanned by
- Ingestion DAG → Discover sources: calls
- Ingestion DAG → Validate and land: maps tasks
- Ingestion DAG → Load raw table: calls
- Discover sources → Validate and land: provides files
- Validate and land → Load raw table: provides Parquet
- Ingestion DAG → Runtime settings: reads configuration
- Ingestion DAG → Warehouse adapter: loads through
- Load raw table → Warehouse adapter: writes raw table
- Ingestion DAG → Data assets: publishes raw asset
- Data assets → Transformation DAG: schedules on raw asset
- Transformation DAG → Warehouse adapter: verifies and reads marts
- Transformation DAG → Runtime settings: reads configuration
- Transformation DAG → dbt models, seeds, snapshots, tests: runs via Cosmos
- dbt models, seeds, snapshots, tests → DuckDB warehouse: reads raw and writes marts
- dbt models, seeds, snapshots, tests → BigQuery warehouse: reads raw and writes marts
- Transformation DAG → Data assets: publishes marts asset
- Data assets → Quality DAG: schedules on marts asset
- Quality DAG → Marts check suite: builds checks from
- Quality DAG → Quality check operator: creates check tasks
- Quality check operator → Check and report engine: runs check
- Quality check operator → Warehouse adapter: queries marts
- Quality DAG → Audit report task: aggregates results
- Audit report task → Check and report engine: uses report types
- Audit report task → Warehouse adapter: appends audit rows
- Quality DAG → Runtime settings: reads configuration
- Warehouse adapter → DuckDB warehouse: connects locally
- Warehouse adapter → BigQuery warehouse: connects in production
- Maintenance DAG → Warehouse adapter: probes and inventories
- Cluster policies → Ingestion DAG: enforces DAG policy
- Cluster policies → Transformation DAG: enforces DAG policy
- Cluster policies → Quality DAG: enforces DAG policy
- Cluster policies → Maintenance DAG: enforces DAG policy