How pashk3r/steam-games-etl-pipeline works
This repository is a Steam-games ETL application. The principal workflow starts in main.py, which creates a Spark session and invokes CSV extraction, PySpark transformation, then JDBC loading to PostgreSQL. Transformation covers the documented cleaning, date/owner splitting, and string-to-array conversions; loading serializes arrays for storage. Configuration supplies runtime settings and logging. Relationships are based on the provided source excerpts and README; no unsampled implementation is assumed.
Main entry points: main.py Pipeline runner · extract.py CSV extraction · load.py JDBC loading · config.py Configuration and logging
Components 9
ETL pipeline
- Pipeline runner
main.py - CSV extraction
src/extract.py - Data transformation
src/transform.py - JDBC loading
src/load.py
Runtime support
- Configuration and logging
src/config.py - ETL exceptions
src/exceptions.py
Other components
- Operator
- Games CSV
- PostgreSQL
Connections 12
- Operator → Pipeline runner: runs
README.md - Pipeline runner → Configuration and logging: sets up
main.py - Pipeline runner → CSV extraction: invokes
main.py - Games CSV → CSV extraction: read and fix
src/extract.py - CSV extraction → Pipeline runner: returns DataFrame
main.py - Pipeline runner → Data transformation: invokes
main.py - Data transformation → Pipeline runner: returns DataFrame
main.py - Pipeline runner → JDBC loading: invokes
main.py - JDBC loading → PostgreSQL: writes via JDBC
src/load.py - CSV extraction → Configuration and logging: reads CSV path
src/extract.py - JDBC loading → Configuration and logging: reads DB settings
src/load.py - Pipeline runner → ETL exceptions: catches ETL errors
main.py