How akhilreddygayam1-cpu/spotify-data-platform works
This is a Spotify music-data lakehouse pipeline: it fetches Spotify search results, stores raw Bronze JSON, normalizes tracks into Silver Parquet, validates Silver data, and aggregates Gold analytical tables. Local Spark scripts and AWS Glue scripts implement parallel transformations; Airflow sequences the Glue jobs. The README documents the principal data stages, while the excerpts show implementation details. Athena queries are present in the tree, but their contents and connections were not sampled, so they are not mapped as runtime components.
Main entry points: silver Silver Parquet · api_ingestion.py API ingestion · spotify_pipeline.py Airflow DAG · transform_silver.py Local Silver transform
Components 16
Spotify ingestion
- Credential config
src/config.py - API ingestion
src/api_ingestion.py - Raw JSON files
data/raw
Lakehouse transformations
- Bronze S3
- Local Silver transform
src/transform_silver.py - Glue Bronze-to-Silver
aws_scripts/spotify_bronze_to_silver.py - Silver Parquet
data/processed/silver
Data quality
- Silver quality checks
aws_scripts/spotify_silver_to_gold_data_quality.py
AWS orchestration
- Airflow DAG
airflow/dags/spotify_pipeline.py
Analytics
- Local Gold transform
src/transform_gold.py - Glue Silver-to-Gold
aws_scripts/spotify_silver_to_gold.py - Gold Parquet
data/processed/gold
Other components
- Pipeline operator
- Spotify API
- AWS Glue
- Analytical reporting
Connections 21
- Pipeline operator → API ingestion: runs ingestion
README.md - API ingestion → Credential config: reads credentials
src/api_ingestion.py - API ingestion → Spotify API: requests tracks
src/api_ingestion.py - API ingestion → Raw JSON files: saves JSON
src/api_ingestion.py - API ingestion → Bronze S3: uploads raw file
src/api_ingestion.py - Raw JSON files → Local Silver transform: reads JSON
README.md - Local Silver transform → Bronze S3: reads Bronze
src/transform_silver.py - Local Silver transform → Silver Parquet: writes Silver
src/transform_silver.py - Glue Bronze-to-Silver → Bronze S3: reads Bronze
aws_scripts/spotify_bronze_to_silver.py - Glue Bronze-to-Silver → Silver Parquet: writes Silver
aws_scripts/spotify_bronze_to_silver.py - Local Silver transform → Silver Parquet: writes local output
README.md - Local Gold transform → Silver Parquet: reads Silver
src/transform_gold.py - Local Gold transform → Gold Parquet: writes aggregates
src/transform_gold.py - Glue Silver-to-Gold → Silver Parquet: reads Silver
aws_scripts/spotify_silver_to_gold.py - Glue Silver-to-Gold → Gold Parquet: writes aggregates
aws_scripts/spotify_silver_to_gold.py - Silver quality checks → Silver Parquet: validates Silver
aws_scripts/spotify_silver_to_gold_data_quality.py - Airflow DAG → AWS Glue: dispatches Glue jobs
airflow/dags/spotify_pipeline.py - Airflow DAG → Glue Bronze-to-Silver: schedules first stage
README.md - Airflow DAG → Silver quality checks: schedules validation
README.md - Airflow DAG → Glue Silver-to-Gold: schedules aggregation
README.md - Gold Parquet → Analytical reporting: supports analysis
README.md