How auriol00/data_engineering_etl works
This repository runs a scheduled used-car ETL workflow: fetch Heycar Trader listings, retain raw payloads, transform them into cleaned staging records, load a PostgreSQL warehouse, and generate CSV and chart reports. Airflow dispatches the four CLI stages sequentially. The sampled source confirms the scraper, normalization helpers, ORM/session layer, and parts of the pipeline CLI; analysis capabilities and staging behavior are also documented in the README.
Main entry points: run_pipeline.py Pipeline CLI · models.py Vehicle warehouse · scraper.py Listing scraper · session.py DB session
Components 15
Orchestration
- Pipeline DAG
dags/heycar_pipeline_dag.py - Pipeline CLI
src/pipeline/run_pipeline.py
Ingest and Transform
- Listing scraper
src/ingest/scraper.py - Field extraction
src/transform/extract_from_trader.py - Type normalization
src/transform/normalize.py - Make/model standardization
src/transform/standardize.py - Cleaned JSONL staging
src/pipeline/run_pipeline.py
Warehouse
- Raw payload log
src/db/models.py - DB session
src/db/session.py - Schema bootstrap
src/db/bootstrap.py - Vehicle warehouse
src/db/models.py
Analysis
- CSV and chart reports
src/pipeline/run_pipeline.py
Other components
- Airflow scheduler
- Heycar Trader API
- Report user
Connections 19
- Airflow scheduler → Pipeline DAG: schedules
- Pipeline DAG → Pipeline CLI: dispatches stages
- Pipeline CLI → Listing scraper: calls extract
- Listing scraper → Heycar Trader API: fetches listings
- Listing scraper → Pipeline CLI: returns listings
- Pipeline CLI → DB session: opens sessions
- DB session → Vehicle warehouse: connects to
- Pipeline CLI → Raw payload log: stores payloads
- Pipeline CLI → Raw payload log: reads unprocessed rows
- Pipeline CLI → Field extraction: calls transform
- Field extraction → Type normalization: parses values
- Pipeline CLI → Make/model standardization: canonicalizes names
- Pipeline CLI → Cleaned JSONL staging: writes and reads
- Pipeline CLI → Vehicle warehouse: upserts records
- Schema bootstrap → DB session: uses engine
- Schema bootstrap → Vehicle warehouse: creates tables
- Pipeline CLI → Vehicle warehouse: queries for analysis
- Pipeline CLI → CSV and chart reports: writes CSV and charts
- Report user → CSV and chart reports: views