# jairajsaraf/rentals-data-pipeline architecture

> Architecture diagram of the GitHub repository jairajsaraf/rentals-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jairajsaraf/rentals-data-pipeline
- Repository: https://github.com/jairajsaraf/rentals-data-pipeline

## Overview

This repository is a weekly Zillow ZORI ETL and analytics pipeline. Airflow coordinates ingestion, PySpark transformation, quality validation, DuckDB loading, and dbt modeling. The graph separates orchestration, processing, validation/storage, and analytics; transform substeps are summarized in their implementing module. The DAG’s download task is explicitly a placeholder, so an actual Zillow-to-S3 transfer is not established. The dbt models are represented from the tree and README; their SQL internals were not sampled.

Main entry points: [`dags/rental_pipeline_dag.py`](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py) (Airflow DAG), [`dbt/models/staging/stg_zori_rent.sql`](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/staging/stg_zori_rent.sql) (ZORI Staging), [`jobs/io_utils.py`](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/io_utils.py) (CSV & Parquet I/O), [`jobs/transform.py`](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/transform.py) (ZORI Transforms)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow_dag["Airflow DAG"]
  node_pipeline_config["Pipeline Config<br/>[pipeline.yaml]"]
end

subgraph group_processing["Data Processing"]
  node_io_utils["CSV &amp; Parquet I/O<br/>[io_utils.py]"]
  node_transform_pipeline["ZORI Transforms<br/>[transform.py]"]
end

subgraph group_quality_storage["Quality &amp; Storage"]
  node_processed_zone[("Partitioned Parquet")]
  node_quality_checks["Quality Checks<br/>[data_quality.py]"]
  node_duckdb_loader["DuckDB Loader"]
  node_duckdb_store[("DuckDB Rental Table")]
end

subgraph group_analytics["Analytics"]
  node_dbt_runner["dbt Run &amp; Test"]
  node_staging_model["ZORI Staging<br/>[stg_zori_rent.sql]"]
  node_rent_trends["Rent Trends Mart"]
  node_state_summary["State Rent Summary"]
end

node_zillow(("Zillow ZORI"))
node_raw_zone[("S3 Raw Zone")]

node_airflow_dag -->|"loads settings"| node_pipeline_config
node_airflow_dag -->|"calls I/O"| node_io_utils
node_airflow_dag -->|"runs transforms"| node_transform_pipeline
node_airflow_dag -->|"runs checks"| node_quality_checks
node_airflow_dag -->|"loads data"| node_duckdb_loader
node_airflow_dag -->|"runs dbt"| node_dbt_runner
node_zillow -.->|"provides CSV"| node_raw_zone
node_airflow_dag -.->|"stages raw path"| node_raw_zone
node_raw_zone -->|"read CSV"| node_io_utils
node_transform_pipeline -->|"transforms data"| node_processed_zone
node_io_utils -->|"writes Parquet"| node_processed_zone
node_processed_zone -->|"validates data"| node_quality_checks
node_processed_zone -->|"reads Parquet"| node_duckdb_loader
node_duckdb_loader -->|"writes table"| node_duckdb_store
node_duckdb_store -->|"models source"| node_dbt_runner
node_dbt_runner -->|"materializes"| node_staging_model
node_dbt_runner -->|"materializes"| node_rent_trends
node_dbt_runner -->|"materializes"| node_state_summary
node_staging_model -.->|"feeds model"| node_rent_trends
node_staging_model -.->|"feeds model"| node_state_summary

click node_airflow_dag "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py"
click node_pipeline_config "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/config/pipeline.yaml"
click node_io_utils "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/io_utils.py"
click node_transform_pipeline "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/transform.py"
click node_quality_checks "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/data_quality.py"
click node_duckdb_loader "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py"
click node_dbt_runner "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py"
click node_staging_model "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/staging/stg_zori_rent.sql"
click node_rent_trends "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/mart/mart_rent_trends.sql"
click node_state_summary "https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/mart/mart_state_rent_summary.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow_dag,node_pipeline_config toneBlue
class node_io_utils,node_transform_pipeline,node_raw_zone toneAmber
class node_processed_zone,node_quality_checks,node_duckdb_loader,node_duckdb_store toneMint
class node_dbt_runner,node_staging_model,node_rent_trends,node_state_summary toneRose
class node_zillow toneIndigo
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py): `dags/rental_pipeline_dag.py`
- [Pipeline Config](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/config/pipeline.yaml): `config/pipeline.yaml`

### Data Processing

- [CSV & Parquet I/O](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/io_utils.py): `jobs/io_utils.py`
- [ZORI Transforms](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/transform.py): `jobs/transform.py`

### Quality & Storage

- **Partitioned Parquet**
- [Quality Checks](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/jobs/data_quality.py): `jobs/data_quality.py`
- [DuckDB Loader](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py): `dags/rental_pipeline_dag.py`
- **DuckDB Rental Table**

### Analytics

- [dbt Run & Test](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dags/rental_pipeline_dag.py): `dags/rental_pipeline_dag.py`
- [ZORI Staging](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/staging/stg_zori_rent.sql): `dbt/models/staging/stg_zori_rent.sql`
- [Rent Trends Mart](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/mart/mart_rent_trends.sql): `dbt/models/mart/mart_rent_trends.sql`
- [State Rent Summary](https://github.com/jairajsaraf/rentals-data-pipeline/blob/master/dbt/models/mart/mart_state_rent_summary.sql): `dbt/models/mart/mart_state_rent_summary.sql`

### Other components

- **Zillow ZORI**
- **S3 Raw Zone**

## Connections

- Airflow DAG → Pipeline Config: loads settings (evidence: `dags/rental_pipeline_dag.py`)
- Airflow DAG → CSV & Parquet I/O: calls I/O (evidence: `dags/rental_pipeline_dag.py`)
- Airflow DAG → ZORI Transforms: runs transforms (evidence: `dags/rental_pipeline_dag.py`)
- Airflow DAG → Quality Checks: runs checks (evidence: `dags/rental_pipeline_dag.py`)
- Airflow DAG → DuckDB Loader: loads data (evidence: `dags/rental_pipeline_dag.py`)
- Airflow DAG → dbt Run & Test: runs dbt (evidence: `dags/rental_pipeline_dag.py`)
- Zillow ZORI → S3 Raw Zone: provides CSV (evidence: `README.md`)
- Airflow DAG → S3 Raw Zone: stages raw path (evidence: `dags/rental_pipeline_dag.py`)
- S3 Raw Zone → CSV & Parquet I/O: read CSV
- ZORI Transforms → Partitioned Parquet: transforms data
- CSV & Parquet I/O → Partitioned Parquet: writes Parquet
- Partitioned Parquet → Quality Checks: validates data
- Partitioned Parquet → DuckDB Loader: reads Parquet (evidence: `dags/rental_pipeline_dag.py`)
- DuckDB Loader → DuckDB Rental Table: writes table (evidence: `dags/rental_pipeline_dag.py`)
- DuckDB Rental Table → dbt Run & Test: models source (evidence: `dags/rental_pipeline_dag.py`)
- dbt Run & Test → ZORI Staging: materializes (evidence: `README.md`)
- dbt Run & Test → Rent Trends Mart: materializes (evidence: `README.md`)
- dbt Run & Test → State Rent Summary: materializes (evidence: `README.md`)
- ZORI Staging → Rent Trends Mart: feeds model
- ZORI Staging → State Rent Summary: feeds model