How ksilogacademy/data-engineer-01-pipeline-api works
This is a monthly NYC TLC Yellow Taxi data pipeline. The user supplies a year and month to main.py, which runs extraction and then transformation as separate subprocesses, stopping if either fails. Extraction pages Socrata results, saves resumable raw pages, and checks the extracted total against the API. Transformation loads those pages, cleans and validates the data, imputes missing values, flags fare discrepancies, and writes a CSV and quality report. The graph separates orchestration, extraction, transformation, and shared configuration. The README documents some transformation details whose implementation is only partially excerpted.
Main entry points: extract.py Page extraction · transform.py Transform runner · main.py Pipeline runner · config.py Pipeline settings
Components 15
Orchestration
- Pipeline runner
src/main.py
Extraction
- Page extraction
src/extract.py - Socrata API
- Raw JSON pages
- Count validation
src/extract.py
Transformation
- Transform runner
src/transform.py - Raw data loader
src/transform.py - Type and deduplicate
src/transform.py - Quality filtering
src/transform.py - Missing-value imputation
src/transform.py - Fare discrepancy flag
src/transform.py - Cleaned CSV
- Quality report
reports/quality_report.json
Shared configuration
- Pipeline settings
src/config.py
Other components
- Pipeline user
Connections 18
- Pipeline user → Pipeline runner: provides month
README.md - Pipeline runner → Page extraction: runs first
src/main.py - Pipeline runner → Transform runner: runs next
src/main.py - Page extraction → Pipeline settings: reads settings
src/extract.py - Transform runner → Pipeline settings: reads settings
src/transform.py - Page extraction → Socrata API: requests pages
src/extract.py - Page extraction → Socrata API: requests count
src/extract.py - Page extraction → Raw JSON pages: writes pages
src/extract.py - Count validation → Socrata API: compares totals
src/extract.py - Page extraction → Count validation: validates total
src/extract.py - Transform runner → Raw data loader: loads data
src/transform.py - Raw data loader → Raw JSON pages: reads pages
src/transform.py - Raw data loader → Type and deduplicate: passes rows
README.md - Type and deduplicate → Quality filtering: passes rows
README.md - Quality filtering → Missing-value imputation: passes rows
README.md - Missing-value imputation → Fare discrepancy flag: enriches rows
README.md - Transform runner → Cleaned CSV: saves result
README.md - Transform runner → Quality report: saves report
src/transform.py