# amidarkcoder/goodreaders_project architecture

> Architecture diagram of the GitHub repository amidarkcoder/goodreaders_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amidarkcoder/goodreaders_project
- Repository: https://github.com/amidarkcoder/goodreaders_project

## Overview

This repository implements a Goodreads data pipeline: generate or collect review and catalog CSVs, move landing data into a working area, transform it with Spark, load Redshift staging and warehouse tables, then build author- and book-focused analytics with Airflow orchestration. The driver and warehouse modules show the central ETL and load calls; the DAG documents analytics tasks. The README is empty, and several referenced modules and configuration files are absent from the tree, so some integrations and analytics wiring are represented only at the documented boundary.

Main entry points: [`goodreads_warehouse_driver.py`](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_warehouse_driver.py) (Warehouse Driver), [`goodreads_etl_dag.py`](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_etl_dag.py) (Airflow DAG), [`goodreads_driver.py`](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_driver.py) (ETL Driver), [`goodreads_transform.py`](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_transform.py) (Spark Transform)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data Ingestion"]
  node_generator["Fake Data Generator"]
  node_csvfiles["Goodreads CSVs"]
  node_zones[("S3 Data Zones<br/>[s3_module.py]")]
end

subgraph group_transform["Spark Transform"]
  node_etl["ETL Driver"]
  node_spark["Spark Transform"]
  node_udf["Spark UDFs<br/>[goodreads_udf.py]"]
  node_processed["Processed Files"]
end

subgraph group_warehouse["Warehouse Loading"]
  node_warehouse_driver["Warehouse Driver"]
  node_staging_sql["Staging SQL"]
  node_upsert_sql["Upsert SQL"]
  node_warehouse_sql["Warehouse SQL"]
  node_redshift[("Redshift Warehouse")]
end

subgraph group_analytics["Analytics"]
  node_scheduler(("Airflow DAG"))
  node_analytics_sql["Analytics Queries"]
  node_analytics["Goodreads Analytics"]
  node_quality["Data Quality Checks<br/>[data_quality.py]"]
end

node_generator -->|"writes CSVs"| node_csvfiles
node_scheduler -->|"SSH dispatches"| node_etl
node_etl -->|"moves and lists"| node_zones
node_zones -->|"supplies working files"| node_spark
node_spark -->|"uses functions"| node_udf
node_spark -->|"writes transformed files"| node_processed
node_etl -->|"runs warehouse load"| node_warehouse_driver
node_warehouse_driver -->|"executes staging SQL"| node_staging_sql
node_warehouse_driver -->|"executes table SQL"| node_warehouse_sql
node_warehouse_driver -->|"executes upserts"| node_upsert_sql
node_warehouse_driver -->|"executes SQL"| node_redshift
node_scheduler -->|"dispatches queries"| node_analytics_sql
node_scheduler -->|"runs analytics tasks"| node_analytics
node_scheduler -->|"runs checks"| node_quality
node_analytics -->|"queries warehouse"| node_redshift
node_quality -->|"checks row counts"| node_redshift

click node_generator "https://github.com/amidarkcoder/goodreaders_project/blob/main/generate_fake_data.py"
click node_scheduler "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_etl_dag.py"
click node_etl "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_driver.py"
click node_zones "https://github.com/amidarkcoder/goodreaders_project/blob/main/s3_module.py"
click node_spark "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_transform.py"
click node_udf "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_udf.py"
click node_warehouse_driver "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_warehouse_driver.py"
click node_staging_sql "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_staging_queries.py"
click node_upsert_sql "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_upsert.py"
click node_warehouse_sql "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_warehouse_queries.py"
click node_analytics_sql "https://github.com/amidarkcoder/goodreaders_project/blob/main/analytics_queries.py"
click node_analytics "https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_analytics.py"
click node_quality "https://github.com/amidarkcoder/goodreaders_project/blob/main/data_quality.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_csvfiles,node_zones toneBlue
class node_etl,node_spark,node_udf,node_processed toneAmber
class node_warehouse_driver,node_staging_sql,node_upsert_sql,node_warehouse_sql,node_redshift toneMint
class node_scheduler,node_analytics_sql,node_analytics,node_quality toneRose
```

## Components

### Data Ingestion

- [Fake Data Generator](https://github.com/amidarkcoder/goodreaders_project/blob/main/generate_fake_data.py): `generate_fake_data.py`
- **Goodreads CSVs**
- [S3 Data Zones](https://github.com/amidarkcoder/goodreaders_project/blob/main/s3_module.py): `s3_module.py`

### Spark Transform

- [ETL Driver](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_driver.py): `goodreads_driver.py`
- [Spark Transform](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_transform.py): `goodreads_transform.py`
- [Spark UDFs](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_udf.py): `goodreads_udf.py`
- **Processed Files**

### Warehouse Loading

- [Warehouse Driver](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_warehouse_driver.py): `goodreads_warehouse_driver.py`
- [Staging SQL](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_staging_queries.py): `goodreads_staging_queries.py`
- [Upsert SQL](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_upsert.py): `goodreads_upsert.py`
- [Warehouse SQL](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_warehouse_queries.py): `goodreads_warehouse_queries.py`
- **Redshift Warehouse**

### Analytics

- [Airflow DAG](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_etl_dag.py): `goodreads_etl_dag.py`
- [Analytics Queries](https://github.com/amidarkcoder/goodreaders_project/blob/main/analytics_queries.py): `analytics_queries.py`
- [Goodreads Analytics](https://github.com/amidarkcoder/goodreaders_project/blob/main/goodreads_analytics.py): `goodreads_analytics.py`
- [Data Quality Checks](https://github.com/amidarkcoder/goodreaders_project/blob/main/data_quality.py): `data_quality.py`

## Connections

- Fake Data Generator → Goodreads CSVs: writes CSVs
- Airflow DAG → ETL Driver: SSH dispatches
- ETL Driver → S3 Data Zones: moves and lists
- S3 Data Zones → Spark Transform: supplies working files
- Spark Transform → Spark UDFs: uses functions
- Spark Transform → Processed Files: writes transformed files
- ETL Driver → Warehouse Driver: runs warehouse load
- Warehouse Driver → Staging SQL: executes staging SQL
- Warehouse Driver → Warehouse SQL: executes table SQL
- Warehouse Driver → Upsert SQL: executes upserts
- Warehouse Driver → Redshift Warehouse: executes SQL
- Airflow DAG → Analytics Queries: dispatches queries
- Airflow DAG → Goodreads Analytics: runs analytics tasks
- Airflow DAG → Data Quality Checks: runs checks
- Goodreads Analytics → Redshift Warehouse: queries warehouse
- Data Quality Checks → Redshift Warehouse: checks row counts