# san089/goodreads_etl_pipeline architecture

> Architecture diagram of the GitHub repository san089/goodreads_etl_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/san089/goodreads_etl_pipeline
- Repository: https://github.com/san089/goodreads_etl_pipeline

## Overview

This repository implements a scheduled Goodreads data pipeline: Goodreads data lands in S3, is copied and transformed, loaded through Redshift staging into warehouse tables, then checked and used to populate analytics tables. Airflow schedules and orders these steps. The README also documents fake-data generation for load testing. The DAG, ETL driver, transformation and warehouse code are sampled; analytics operator and query implementation details are represented only at the documented subsystem level.

Main entry points: [`src/goodreads_driver.py`](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_driver.py) (ETL driver), [`airflow/dags/goodreads_etl_dag.py`](https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/dags/goodreads_etl_dag.py) (Pipeline DAG), [`src/warehouse/goodreads_warehouse_driver.py`](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_warehouse_driver.py) (Warehouse driver), [`src/s3_module.py`](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/s3_module.py) (S3 mover)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration and quality"]
  node_dag["Pipeline DAG"]
  node_quality["Data quality operator<br/>[data_quality.py]"]
  node_analytics_operator["Analytics operator"]
end

subgraph group_etl["ETL and storage"]
  node_landing[("Landing bucket")]
  node_working[("Working bucket")]
  node_processed[("Processed bucket")]
  node_driver["ETL driver"]
  node_s3["S3 mover<br/>[s3_module.py]"]
  node_transform["Spark transforms"]
  node_udf["Transform UDFs<br/>[goodreads_udf.py]"]
end

subgraph group_warehouse["Warehouse and analytics"]
  node_warehouse_driver["Warehouse driver"]
  node_staging_sql["Staging SQL"]
  node_warehouse_sql["Warehouse SQL"]
  node_upsert_sql["Upsert SQL"]
  node_analytics_queries["Analytics queries"]
end

subgraph group_loadtest["Load testing"]
  node_faker["Fake data generator"]
  node_csv_data["Generated CSV data"]
end

node_goodreads_api(("Goodreads API"))
node_redshift[("Redshift")]

node_goodreads_api -.->|"lands data"| node_landing
node_dag -->|"SSH dispatches"| node_driver
node_driver -->|"moves files"| node_s3
node_s3 -->|"reads files"| node_landing
node_s3 -->|"copies files"| node_working
node_driver -->|"lists files"| node_s3
node_driver -->|"cleans processed"| node_s3
node_driver -->|"runs transforms"| node_transform
node_transform -->|"reads datasets"| node_working
node_transform -->|"writes datasets"| node_processed
node_transform -->|"applies functions"| node_udf
node_driver -->|"runs warehouse load"| node_warehouse_driver
node_warehouse_driver -->|"executes statements"| node_staging_sql
node_warehouse_driver -->|"executes statements"| node_warehouse_sql
node_warehouse_driver -->|"executes statements"| node_upsert_sql
node_warehouse_driver -->|"creates and loads tables"| node_redshift
node_redshift -->|"COPY reads data"| node_processed
node_dag -->|"dispatches checks"| node_quality
node_quality -->|"checks table counts"| node_redshift
node_dag -->|"dispatches analytics"| node_analytics_operator
node_dag -->|"supplies queries"| node_analytics_queries
node_analytics_operator -->|"executes queries"| node_redshift
node_dag -->|"dispatches analytics checks"| node_quality
node_faker -->|"writes CSVs"| node_csv_data

click node_dag "https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/dags/goodreads_etl_dag.py"
click node_driver "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_driver.py"
click node_s3 "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/s3_module.py"
click node_transform "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_transform.py"
click node_udf "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_udf.py"
click node_warehouse_driver "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_warehouse_driver.py"
click node_staging_sql "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_staging_queries.py"
click node_warehouse_sql "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_warehouse_queries.py"
click node_upsert_sql "https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_upsert.py"
click node_quality "https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/operators/data_quality.py"
click node_analytics_operator "https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/operators/goodreads_analytics.py"
click node_analytics_queries "https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/helpers/analytics_queries.py"
click node_faker "https://github.com/san089/goodreads_etl_pipeline/blob/master/goodreadsfaker/generate_fake_data.py"
click node_csv_data "https://github.com/san089/goodreads_etl_pipeline/tree/master/SampleData"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag,node_quality,node_analytics_operator toneBlue
class node_landing,node_working,node_processed,node_driver,node_s3,node_transform,node_udf,node_redshift toneAmber
class node_warehouse_driver,node_staging_sql,node_warehouse_sql,node_upsert_sql,node_analytics_queries,node_goodreads_api toneMint
class node_faker,node_csv_data toneRose
```

## Components

### Orchestration and quality

- [Pipeline DAG](https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/dags/goodreads_etl_dag.py): `airflow/dags/goodreads_etl_dag.py`
- [Data quality operator](https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/operators/data_quality.py): `airflow/plugins/operators/data_quality.py`
- [Analytics operator](https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/operators/goodreads_analytics.py): `airflow/plugins/operators/goodreads_analytics.py`

### ETL and storage

- **Landing bucket**
- **Working bucket**
- **Processed bucket**
- [ETL driver](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_driver.py): `src/goodreads_driver.py`
- [S3 mover](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/s3_module.py): `src/s3_module.py`
- [Spark transforms](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_transform.py): `src/goodreads_transform.py`
- [Transform UDFs](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/goodreads_udf.py): `src/goodreads_udf.py`

### Warehouse and analytics

- [Warehouse driver](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_warehouse_driver.py): `src/warehouse/goodreads_warehouse_driver.py`
- [Staging SQL](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_staging_queries.py): `src/warehouse/goodreads_staging_queries.py`
- [Warehouse SQL](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_warehouse_queries.py): `src/warehouse/goodreads_warehouse_queries.py`
- [Upsert SQL](https://github.com/san089/goodreads_etl_pipeline/blob/master/src/warehouse/goodreads_upsert.py): `src/warehouse/goodreads_upsert.py`
- [Analytics queries](https://github.com/san089/goodreads_etl_pipeline/blob/master/airflow/plugins/helpers/analytics_queries.py): `airflow/plugins/helpers/analytics_queries.py`

### Load testing

- [Fake data generator](https://github.com/san089/goodreads_etl_pipeline/blob/master/goodreadsfaker/generate_fake_data.py): `goodreadsfaker/generate_fake_data.py`
- [Generated CSV data](https://github.com/san089/goodreads_etl_pipeline/tree/master/SampleData): `SampleData`

### Other components

- **Goodreads API**
- **Redshift**

## Connections

- Goodreads API → Landing bucket: lands data
- Pipeline DAG → ETL driver: SSH dispatches
- ETL driver → S3 mover: moves files
- S3 mover → Landing bucket: reads files
- S3 mover → Working bucket: copies files
- ETL driver → S3 mover: lists files
- ETL driver → S3 mover: cleans processed
- ETL driver → Spark transforms: runs transforms
- Spark transforms → Working bucket: reads datasets
- Spark transforms → Processed bucket: writes datasets
- Spark transforms → Transform UDFs: applies functions
- ETL driver → Warehouse driver: runs warehouse load
- Warehouse driver → Staging SQL: executes statements
- Warehouse driver → Warehouse SQL: executes statements
- Warehouse driver → Upsert SQL: executes statements
- Warehouse driver → Redshift: creates and loads tables
- Redshift → Processed bucket: COPY reads data
- Pipeline DAG → Data quality operator: dispatches checks
- Data quality operator → Redshift: checks table counts
- Pipeline DAG → Analytics operator: dispatches analytics
- Pipeline DAG → Analytics queries: supplies queries
- Analytics operator → Redshift: executes queries
- Pipeline DAG → Data quality operator: dispatches analytics checks
- Fake data generator → Generated CSV data: writes CSVs