# auriol00/data_engineering_etl architecture

> Architecture diagram of the GitHub repository auriol00/data_engineering_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/auriol00/data_engineering_etl
- Repository: https://github.com/auriol00/data_engineering_etl

## Overview

This repository runs a scheduled used-car ETL workflow: fetch Heycar Trader listings, retain raw payloads, transform them into cleaned staging records, load a PostgreSQL warehouse, and generate CSV and chart reports. Airflow dispatches the four CLI stages sequentially. The sampled source confirms the scraper, normalization helpers, ORM/session layer, and parts of the pipeline CLI; analysis capabilities and staging behavior are also documented in the README.

Main entry points: [`src/pipeline/run_pipeline.py`](https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py) (Pipeline CLI), [`src/db/models.py`](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/models.py) (Vehicle warehouse), [`src/ingest/scraper.py`](https://github.com/auriol00/data_engineering_etl/blob/main/src/ingest/scraper.py) (Listing scraper), [`src/db/session.py`](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/session.py) (DB session)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_dag["Pipeline DAG"]
  node_cli["Pipeline CLI<br/>[run_pipeline.py]"]
end

subgraph group_ingest["Ingest and Transform"]
  node_scraper["Listing scraper<br/>[scraper.py]"]
  node_extractor["Field extraction"]
  node_normalizer["Type normalization<br/>[normalize.py]"]
  node_standardizer["Make/model standardization<br/>[standardize.py]"]
  node_staging["Cleaned JSONL staging<br/>[run_pipeline.py]"]
end

subgraph group_warehouse["Warehouse"]
  node_raw[("Raw payload log<br/>[models.py]")]
  node_session["DB session<br/>[session.py]"]
  node_bootstrap["Schema bootstrap<br/>[bootstrap.py]"]
  node_warehouse_db[("Vehicle warehouse<br/>[models.py]")]
end

subgraph group_analysis["Analysis"]
  node_reports["CSV and chart reports<br/>[run_pipeline.py]"]
end

node_scheduler(("Airflow scheduler"))
node_trader(("Heycar Trader API"))
node_analyst(("Report user"))

node_scheduler -->|"schedules"| node_dag
node_dag -->|"dispatches stages"| node_cli
node_cli -->|"calls extract"| node_scraper
node_scraper -->|"fetches listings"| node_trader
node_scraper -->|"returns listings"| node_cli
node_cli -->|"opens sessions"| node_session
node_session -->|"connects to"| node_warehouse_db
node_cli -->|"stores payloads"| node_raw
node_cli -->|"reads unprocessed rows"| node_raw
node_cli -->|"calls transform"| node_extractor
node_extractor -->|"parses values"| node_normalizer
node_cli -->|"canonicalizes names"| node_standardizer
node_cli -->|"writes and reads"| node_staging
node_cli -->|"upserts records"| node_warehouse_db
node_bootstrap -->|"uses engine"| node_session
node_bootstrap -->|"creates tables"| node_warehouse_db
node_cli -->|"queries for analysis"| node_warehouse_db
node_cli -->|"writes CSV and charts"| node_reports
node_analyst -->|"views"| node_reports

click node_dag "https://github.com/auriol00/data_engineering_etl/blob/main/dags/heycar_pipeline_dag.py"
click node_cli "https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py"
click node_scraper "https://github.com/auriol00/data_engineering_etl/blob/main/src/ingest/scraper.py"
click node_raw "https://github.com/auriol00/data_engineering_etl/blob/main/src/db/models.py"
click node_extractor "https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/extract_from_trader.py"
click node_normalizer "https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/normalize.py"
click node_standardizer "https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/standardize.py"
click node_staging "https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py"
click node_session "https://github.com/auriol00/data_engineering_etl/blob/main/src/db/session.py"
click node_bootstrap "https://github.com/auriol00/data_engineering_etl/blob/main/src/db/bootstrap.py"
click node_warehouse_db "https://github.com/auriol00/data_engineering_etl/blob/main/src/db/models.py"
click node_reports "https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag,node_cli,node_analyst toneBlue
class node_scraper,node_extractor,node_normalizer,node_standardizer,node_staging toneAmber
class node_raw,node_session,node_bootstrap,node_warehouse_db,node_trader toneMint
class node_reports,node_scheduler toneRose
```

## Components

### Orchestration

- [Pipeline DAG](https://github.com/auriol00/data_engineering_etl/blob/main/dags/heycar_pipeline_dag.py): `dags/heycar_pipeline_dag.py`
- [Pipeline CLI](https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py): `src/pipeline/run_pipeline.py`

### Ingest and Transform

- [Listing scraper](https://github.com/auriol00/data_engineering_etl/blob/main/src/ingest/scraper.py): `src/ingest/scraper.py`
- [Field extraction](https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/extract_from_trader.py): `src/transform/extract_from_trader.py`
- [Type normalization](https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/normalize.py): `src/transform/normalize.py`
- [Make/model standardization](https://github.com/auriol00/data_engineering_etl/blob/main/src/transform/standardize.py): `src/transform/standardize.py`
- [Cleaned JSONL staging](https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py): `src/pipeline/run_pipeline.py`

### Warehouse

- [Raw payload log](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/models.py): `src/db/models.py`
- [DB session](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/session.py): `src/db/session.py`
- [Schema bootstrap](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/bootstrap.py): `src/db/bootstrap.py`
- [Vehicle warehouse](https://github.com/auriol00/data_engineering_etl/blob/main/src/db/models.py): `src/db/models.py`

### Analysis

- [CSV and chart reports](https://github.com/auriol00/data_engineering_etl/blob/main/src/pipeline/run_pipeline.py): `src/pipeline/run_pipeline.py`

### Other components

- **Airflow scheduler**
- **Heycar Trader API**
- **Report user**

## Connections

- Airflow scheduler → Pipeline DAG: schedules
- Pipeline DAG → Pipeline CLI: dispatches stages
- Pipeline CLI → Listing scraper: calls extract
- Listing scraper → Heycar Trader API: fetches listings
- Listing scraper → Pipeline CLI: returns listings
- Pipeline CLI → DB session: opens sessions
- DB session → Vehicle warehouse: connects to
- Pipeline CLI → Raw payload log: stores payloads
- Pipeline CLI → Raw payload log: reads unprocessed rows
- Pipeline CLI → Field extraction: calls transform
- Field extraction → Type normalization: parses values
- Pipeline CLI → Make/model standardization: canonicalizes names
- Pipeline CLI → Cleaned JSONL staging: writes and reads
- Pipeline CLI → Vehicle warehouse: upserts records
- Schema bootstrap → DB session: uses engine
- Schema bootstrap → Vehicle warehouse: creates tables
- Pipeline CLI → Vehicle warehouse: queries for analysis
- Pipeline CLI → CSV and chart reports: writes CSV and charts
- Report user → CSV and chart reports: views