How amidarkcoder/goodreaders_project works
This repository implements a Goodreads data pipeline: generate or collect review and catalog CSVs, move landing data into a working area, transform it with Spark, load Redshift staging and warehouse tables, then build author- and book-focused analytics with Airflow orchestration. The driver and warehouse modules show the central ETL and load calls; the DAG documents analytics tasks. The README is empty, and several referenced modules and configuration files are absent from the tree, so some integrations and analytics wiring are represented only at the documented boundary.
Main entry points: goodreads_warehouse_driver.py Warehouse Driver · goodreads_etl_dag.py Airflow DAG · goodreads_driver.py ETL Driver · goodreads_transform.py Spark Transform
Components 16
Data Ingestion
- Fake Data Generator
generate_fake_data.py - Goodreads CSVs
- S3 Data Zones
s3_module.py
Spark Transform
- ETL Driver
goodreads_driver.py - Spark Transform
goodreads_transform.py - Spark UDFs
goodreads_udf.py - Processed Files
Warehouse Loading
- Warehouse Driver
goodreads_warehouse_driver.py - Staging SQL
goodreads_staging_queries.py - Upsert SQL
goodreads_upsert.py - Warehouse SQL
goodreads_warehouse_queries.py - Redshift Warehouse
Analytics
- Airflow DAG
goodreads_etl_dag.py - Analytics Queries
analytics_queries.py - Goodreads Analytics
goodreads_analytics.py - Data Quality Checks
data_quality.py
Connections 16
- Fake Data Generator → Goodreads CSVs: writes CSVs
- Airflow DAG → ETL Driver: SSH dispatches
- ETL Driver → S3 Data Zones: moves and lists
- S3 Data Zones → Spark Transform: supplies working files
- Spark Transform → Spark UDFs: uses functions
- Spark Transform → Processed Files: writes transformed files
- ETL Driver → Warehouse Driver: runs warehouse load
- Warehouse Driver → Staging SQL: executes staging SQL
- Warehouse Driver → Warehouse SQL: executes table SQL
- Warehouse Driver → Upsert SQL: executes upserts
- Warehouse Driver → Redshift Warehouse: executes SQL
- Airflow DAG → Analytics Queries: dispatches queries
- Airflow DAG → Goodreads Analytics: runs analytics tasks
- Airflow DAG → Data Quality Checks: runs checks
- Goodreads Analytics → Redshift Warehouse: queries warehouse
- Data Quality Checks → Redshift Warehouse: checks row counts