How san089/goodreads_etl_pipeline works
This repository implements a scheduled Goodreads data pipeline: Goodreads data lands in S3, is copied and transformed, loaded through Redshift staging into warehouse tables, then checked and used to populate analytics tables. Airflow schedules and orders these steps. The README also documents fake-data generation for load testing. The DAG, ETL driver, transformation and warehouse code are sampled; analytics operator and query implementation details are represented only at the documented subsystem level.
Main entry points: goodreads_driver.py ETL driver · goodreads_etl_dag.py Pipeline DAG · goodreads_warehouse_driver.py Warehouse driver · s3_module.py S3 mover
Components 19
Orchestration and quality
- Pipeline DAG
airflow/dags/goodreads_etl_dag.py - Data quality operator
airflow/plugins/operators/data_quality.py - Analytics operator
airflow/plugins/operators/goodreads_analytics.py
ETL and storage
- Landing bucket
- Working bucket
- Processed bucket
- ETL driver
src/goodreads_driver.py - S3 mover
src/s3_module.py - Spark transforms
src/goodreads_transform.py - Transform UDFs
src/goodreads_udf.py
Warehouse and analytics
- Warehouse driver
src/warehouse/goodreads_warehouse_driver.py - Staging SQL
src/warehouse/goodreads_staging_queries.py - Warehouse SQL
src/warehouse/goodreads_warehouse_queries.py - Upsert SQL
src/warehouse/goodreads_upsert.py - Analytics queries
airflow/plugins/helpers/analytics_queries.py
Load testing
- Fake data generator
goodreadsfaker/generate_fake_data.py - Generated CSV data
SampleData
Other components
- Goodreads API
- Redshift
Connections 24
- Goodreads API → Landing bucket: lands data
- Pipeline DAG → ETL driver: SSH dispatches
- ETL driver → S3 mover: moves files
- S3 mover → Landing bucket: reads files
- S3 mover → Working bucket: copies files
- ETL driver → S3 mover: lists files
- ETL driver → S3 mover: cleans processed
- ETL driver → Spark transforms: runs transforms
- Spark transforms → Working bucket: reads datasets
- Spark transforms → Processed bucket: writes datasets
- Spark transforms → Transform UDFs: applies functions
- ETL driver → Warehouse driver: runs warehouse load
- Warehouse driver → Staging SQL: executes statements
- Warehouse driver → Warehouse SQL: executes statements
- Warehouse driver → Upsert SQL: executes statements
- Warehouse driver → Redshift: creates and loads tables
- Redshift → Processed bucket: COPY reads data
- Pipeline DAG → Data quality operator: dispatches checks
- Data quality operator → Redshift: checks table counts
- Pipeline DAG → Analytics operator: dispatches analytics
- Pipeline DAG → Analytics queries: supplies queries
- Analytics operator → Redshift: executes queries
- Pipeline DAG → Data quality operator: dispatches analytics checks
- Fake data generator → Generated CSV data: writes CSVs