# akhilreddygayam1-cpu/spotify-data-platform architecture

> Architecture diagram of the GitHub repository akhilreddygayam1-cpu/spotify-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akhilreddygayam1-cpu/spotify-data-platform
- Repository: https://github.com/akhilreddygayam1-cpu/spotify-data-platform

## Overview

This is a Spotify music-data lakehouse pipeline: it fetches Spotify search results, stores raw Bronze JSON, normalizes tracks into Silver Parquet, validates Silver data, and aggregates Gold analytical tables. Local Spark scripts and AWS Glue scripts implement parallel transformations; Airflow sequences the Glue jobs. The README documents the principal data stages, while the excerpts show implementation details. Athena queries are present in the tree, but their contents and connections were not sampled, so they are not mapped as runtime components.

Main entry points: [`data/processed/silver`](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/processed/silver) (Silver Parquet), [`src/api_ingestion.py`](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/api_ingestion.py) (API ingestion), [`airflow/dags/spotify_pipeline.py`](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/airflow/dags/spotify_pipeline.py) (Airflow DAG), [`src/transform_silver.py`](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/transform_silver.py) (Local Silver transform)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Spotify ingestion"]
  node_config["Credential config<br/>[config.py]"]
  node_ingest["API ingestion<br/>[api_ingestion.py]"]
  node_raw_local[("Raw JSON files")]
end

subgraph group_lakehouse["Lakehouse transformations"]
  node_bronze_s3[("Bronze S3")]
  node_local_silver["Local Silver transform"]
  node_glue_silver["Glue Bronze-to-Silver"]
  node_silver_s3[("Silver Parquet")]
end

subgraph group_quality["Data quality"]
  node_quality["Silver quality checks"]
end

subgraph group_orchestration["AWS orchestration"]
  node_airflow["Airflow DAG"]
end

subgraph group_analytics["Analytics"]
  node_local_gold["Local Gold transform<br/>[transform_gold.py]"]
  node_glue_gold["Glue Silver-to-Gold"]
  node_gold_data[("Gold Parquet")]
end

node_spotify_user(("Pipeline operator"))
node_spotify_api["Spotify API"]
node_glue_service["AWS Glue"]
node_reports(("Analytical reporting"))

node_spotify_user -->|"runs ingestion"| node_ingest
node_ingest -->|"reads credentials"| node_config
node_ingest -->|"requests tracks"| node_spotify_api
node_ingest -->|"saves JSON"| node_raw_local
node_ingest -->|"uploads raw file"| node_bronze_s3
node_raw_local -->|"reads JSON"| node_local_silver
node_local_silver -->|"reads Bronze"| node_bronze_s3
node_local_silver -->|"writes Silver"| node_silver_s3
node_glue_silver -->|"reads Bronze"| node_bronze_s3
node_glue_silver -->|"writes Silver"| node_silver_s3
node_local_silver -->|"writes local output"| node_silver_s3
node_local_gold -->|"reads Silver"| node_silver_s3
node_local_gold -->|"writes aggregates"| node_gold_data
node_glue_gold -->|"reads Silver"| node_silver_s3
node_glue_gold -->|"writes aggregates"| node_gold_data
node_quality -->|"validates Silver"| node_silver_s3
node_airflow -->|"dispatches Glue jobs"| node_glue_service
node_airflow -->|"schedules first stage"| node_glue_silver
node_airflow -->|"schedules validation"| node_quality
node_airflow -->|"schedules aggregation"| node_glue_gold
node_gold_data -->|"supports analysis"| node_reports

click node_config "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/config.py"
click node_ingest "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/api_ingestion.py"
click node_raw_local "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/raw"
click node_local_silver "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/transform_silver.py"
click node_glue_silver "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_bronze_to_silver.py"
click node_silver_s3 "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/processed/silver"
click node_local_gold "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/transform_gold.py"
click node_glue_gold "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_silver_to_gold.py"
click node_gold_data "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/processed/gold"
click node_quality "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_silver_to_gold_data_quality.py"
click node_airflow "https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/airflow/dags/spotify_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_ingest,node_raw_local toneBlue
class node_bronze_s3,node_local_silver,node_glue_silver,node_silver_s3 toneAmber
class node_quality,node_spotify_api toneMint
class node_airflow toneRose
class node_local_gold,node_glue_gold,node_gold_data,node_spotify_user,node_glue_service,node_reports toneIndigo
```

## Components

### Spotify ingestion

- [Credential config](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/config.py): `src/config.py`
- [API ingestion](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/api_ingestion.py): `src/api_ingestion.py`
- [Raw JSON files](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/raw): `data/raw`

### Lakehouse transformations

- **Bronze S3**
- [Local Silver transform](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/transform_silver.py): `src/transform_silver.py`
- [Glue Bronze-to-Silver](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_bronze_to_silver.py): `aws_scripts/spotify_bronze_to_silver.py`
- [Silver Parquet](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/processed/silver): `data/processed/silver`

### Data quality

- [Silver quality checks](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_silver_to_gold_data_quality.py): `aws_scripts/spotify_silver_to_gold_data_quality.py`

### AWS orchestration

- [Airflow DAG](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/airflow/dags/spotify_pipeline.py): `airflow/dags/spotify_pipeline.py`

### Analytics

- [Local Gold transform](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/src/transform_gold.py): `src/transform_gold.py`
- [Glue Silver-to-Gold](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/blob/main/aws_scripts/spotify_silver_to_gold.py): `aws_scripts/spotify_silver_to_gold.py`
- [Gold Parquet](https://github.com/akhilreddygayam1-cpu/spotify-data-platform/tree/main/data/processed/gold): `data/processed/gold`

### Other components

- **Pipeline operator**
- **Spotify API**
- **AWS Glue**
- **Analytical reporting**

## Connections

- Pipeline operator → API ingestion: runs ingestion (evidence: `README.md`)
- API ingestion → Credential config: reads credentials (evidence: `src/api_ingestion.py`)
- API ingestion → Spotify API: requests tracks (evidence: `src/api_ingestion.py`)
- API ingestion → Raw JSON files: saves JSON (evidence: `src/api_ingestion.py`)
- API ingestion → Bronze S3: uploads raw file (evidence: `src/api_ingestion.py`)
- Raw JSON files → Local Silver transform: reads JSON (evidence: `README.md`)
- Local Silver transform → Bronze S3: reads Bronze (evidence: `src/transform_silver.py`)
- Local Silver transform → Silver Parquet: writes Silver (evidence: `src/transform_silver.py`)
- Glue Bronze-to-Silver → Bronze S3: reads Bronze (evidence: `aws_scripts/spotify_bronze_to_silver.py`)
- Glue Bronze-to-Silver → Silver Parquet: writes Silver (evidence: `aws_scripts/spotify_bronze_to_silver.py`)
- Local Silver transform → Silver Parquet: writes local output (evidence: `README.md`)
- Local Gold transform → Silver Parquet: reads Silver (evidence: `src/transform_gold.py`)
- Local Gold transform → Gold Parquet: writes aggregates (evidence: `src/transform_gold.py`)
- Glue Silver-to-Gold → Silver Parquet: reads Silver (evidence: `aws_scripts/spotify_silver_to_gold.py`)
- Glue Silver-to-Gold → Gold Parquet: writes aggregates (evidence: `aws_scripts/spotify_silver_to_gold.py`)
- Silver quality checks → Silver Parquet: validates Silver (evidence: `aws_scripts/spotify_silver_to_gold_data_quality.py`)
- Airflow DAG → AWS Glue: dispatches Glue jobs (evidence: `airflow/dags/spotify_pipeline.py`)
- Airflow DAG → Glue Bronze-to-Silver: schedules first stage (evidence: `README.md`)
- Airflow DAG → Silver quality checks: schedules validation (evidence: `README.md`)
- Airflow DAG → Glue Silver-to-Gold: schedules aggregation (evidence: `README.md`)
- Gold Parquet → Analytical reporting: supports analysis (evidence: `README.md`)
