How jairajsaraf/rentals-data-pipeline works
This repository is a weekly Zillow ZORI ETL and analytics pipeline. Airflow coordinates ingestion, PySpark transformation, quality validation, DuckDB loading, and dbt modeling. The graph separates orchestration, processing, validation/storage, and analytics; transform substeps are summarized in their implementing module. The DAG’s download task is explicitly a placeholder, so an actual Zillow-to-S3 transfer is not established. The dbt models are represented from the tree and README; their SQL internals were not sampled.
Main entry points: rental_pipeline_dag.py Airflow DAG · stg_zori_rent.sql ZORI Staging · io_utils.py CSV & Parquet I/O · transform.py ZORI Transforms
Components 14
Orchestration
- Airflow DAG
dags/rental_pipeline_dag.py - Pipeline Config
config/pipeline.yaml
Data Processing
- CSV & Parquet I/O
jobs/io_utils.py - ZORI Transforms
jobs/transform.py
Quality & Storage
- Partitioned Parquet
- Quality Checks
jobs/data_quality.py - DuckDB Loader
dags/rental_pipeline_dag.py - DuckDB Rental Table
Analytics
- dbt Run & Test
dags/rental_pipeline_dag.py - ZORI Staging
dbt/models/staging/stg_zori_rent.sql - Rent Trends Mart
dbt/models/mart/mart_rent_trends.sql - State Rent Summary
dbt/models/mart/mart_state_rent_summary.sql
Other components
- Zillow ZORI
- S3 Raw Zone
Connections 20
- Airflow DAG → Pipeline Config: loads settings
dags/rental_pipeline_dag.py - Airflow DAG → CSV & Parquet I/O: calls I/O
dags/rental_pipeline_dag.py - Airflow DAG → ZORI Transforms: runs transforms
dags/rental_pipeline_dag.py - Airflow DAG → Quality Checks: runs checks
dags/rental_pipeline_dag.py - Airflow DAG → DuckDB Loader: loads data
dags/rental_pipeline_dag.py - Airflow DAG → dbt Run & Test: runs dbt
dags/rental_pipeline_dag.py - Zillow ZORI → S3 Raw Zone: provides CSV
README.md - Airflow DAG → S3 Raw Zone: stages raw path
dags/rental_pipeline_dag.py - S3 Raw Zone → CSV & Parquet I/O: read CSV
- ZORI Transforms → Partitioned Parquet: transforms data
- CSV & Parquet I/O → Partitioned Parquet: writes Parquet
- Partitioned Parquet → Quality Checks: validates data
- Partitioned Parquet → DuckDB Loader: reads Parquet
dags/rental_pipeline_dag.py - DuckDB Loader → DuckDB Rental Table: writes table
dags/rental_pipeline_dag.py - DuckDB Rental Table → dbt Run & Test: models source
dags/rental_pipeline_dag.py - dbt Run & Test → ZORI Staging: materializes
README.md - dbt Run & Test → Rent Trends Mart: materializes
README.md - dbt Run & Test → State Rent Summary: materializes
README.md - ZORI Staging → Rent Trends Mart: feeds model
- ZORI Staging → State Rent Summary: feeds model