# ayushigupta9723/spotify-recommender-system architecture

> Architecture diagram of the GitHub repository ayushigupta9723/spotify-recommender-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushigupta9723/spotify-recommender-system
- Repository: https://github.com/ayushigupta9723/spotify-recommender-system

## Overview

##### Project Type and Purpose
- Type: A data-science / ML pipeline for music recommendation (content-based, collaborative, hybrid).
- Purpose: Ingest raw music and user data, clean & transform it, engineer features, train/evaluate models, and deploy an inference API (via app.py).

##### Main Components
a. Data Sources & Storage
– “external” data store (3rd-party CSV)
– “raw” data folder (immutable dumps under DVC control)
– “interim” (transformed but not final)
– “processed” (final datasets ready for modeling)
– DVC remote (could be S3/GCS for versioned artifacts)

b. Pipeline Orchestrator
– DVC (dvc.yaml, dvc.lock) drives the stages: data download → cleaning → feature extraction → model training → evaluation → export.
– Makefile front-ends common commands (`make data`, `make train`, etc.).

c. Data-Processing & Feature Engineering Modules
– src/data/data_cleaning.py
– src/features/content_based_filtering.py
– src/features/collabrative_filtering.py

d. Model Training & Inference
– src/models/hybrid_recommendation.py (combines content + collaborative)
– src/models/transform.py (data transforms)
– transformer.joblib (saved preprocessing pipeline)
– models/ (folder for serialized models, predictions)

e. Visualization & Reporting
– reports/figures and docs/ (Sphinx docs + notebooks/ for EDA)

f. Deployment & API
– app.py (Flask/FastAPI script exposing endpoints for recommendations)
– Dockerfile (containerizes app + pipeline environment)

g. CI/CD & Testing
– .github/workflows/ci.yaml (runs tests, lint, dvc repro checks)
– test_environment.py, tox.ini (test harness, environment validation)

##### Relationships & Data Flow
1. Raw CSV data (data/external, data/raw) → DVC pulls it in
2. data_cleaning.py → writes to data/interim
3. feature scripts → consume interim → produce data/processed or feature artifacts
4. model training (hybrid_recommendation) → reads features → outputs model artifacts into models/ + transformer.joblib
5. Visualization scripts & Jupyter notebooks read processed data + model outputs → generate reports & figures
6. app.py loads transformer.joblib + model files → serves online recommendations
7. CI pipeline validates code, runs dvc repro, tests, builds Docker image, and can push to registry

##### Architectural Patterns & Principles
– Modular pipeline stages (single responsibility per script folder)
– Versioned data + artifacts via DVC for reproducibility
– Clear separation: Data layer, Processing/Modeling layer, Serving layer, CI/CD
– Containerization for environment consistency (Docker)
– Infrastructure-as-code style orchestration (Make + DVC + GitHub Actions)

– Rectangles for code modules (data cleaning, feature eng, modeling, visualization, app)
– GitHub logo for CI/CD workflow

– Data Layer (all data folders + DVC remote)
– Processing Layer (src/data, src/features)
– Modeling Layer (src/models + models/)
– Serving Layer (app.py, Dockerfile)
– Reporting Layer (notebooks, src/visualization, docs, reports)
– Orchestration Layer (Makefile, dvc.yaml, GitHub Actions)

– Label every data store and pipeline stage clearly.
– Don’t forget external dependencies: GitHub Actions, Docker registry, any external data sources or APIs.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        ExternalData[(External Data)]:::data
        click ExternalData "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/data/external"
        RawData[(Raw Data)]:::data
        click RawData "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/data/raw"
        InterimData[(Interim Data)]:::data
        click InterimData "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/data/interim"
        ProcessedData[(Processed Data)]:::data
        click ProcessedData "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/data/processed"
        DVCRemote[(DVC Remote)]:::data
        click DVCRemote "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/.dvc/config"
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        DataCleaning["Data Cleaning Module"]:::processing
        click DataCleaning "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/data/data_cleaning.py"
        ContentFE["Content-Based Filtering"]:::processing
        click ContentFE "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/features/content_based_filtering.py"
        CollabFE["Collaborative Filtering"]:::processing
        click CollabFE "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/features/collabrative_filtering.py"
    end

    %% Modeling Layer
    subgraph "Modeling Layer"
        direction TB
        HybridModel["Hybrid Recommendation Training"]:::modeling
        click HybridModel "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/models/hybrid_recommendation.py"
        DataTransform["Data Transform Module"]:::modeling
        click DataTransform "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/models/transform.py"
        TransformerArtifact["Transformer Artifact<br/>transformer.joblib"]:::modeling
        click TransformerArtifact "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/transformer.joblib"
        ModelArtifacts[(Serialized Models)]:::modeling
        click ModelArtifacts "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/models/"
    end

    %% Reporting Layer
    subgraph "Reporting Layer"
        direction TB
        Visualization["Visualization Scripts"]:::reporting
        click Visualization "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/src/visualization/visualize.py"
        Reports[(Figures Directory)]:::reporting
        click Reports "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/reports/figures"
        Docs[(Documentation)]:::reporting
        click Docs "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/docs/"
        Notebooks[(Exploratory Notebooks)]:::reporting
        click Notebooks "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/notebooks/"
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        APIApp["API Application<br/>app.py"]:::serving
        click APIApp "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/app.py"
        DockerDef["Containerization Definition"]:::serving
        click DockerDef "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/Dockerfile"
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        Makefile["Makefile"]:::orchestration
        click Makefile "https://github.com/ayushigupta9723/spotify-recommender-system/tree/main/Makefile"
        DVCPipeline["DVC Pipeline<br/>dvc.yaml"]:::orchestration
        click DVCPipeline "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/dvc.yaml"
        DVCLock["DVC Lock File"]:::orchestration
        click DVCLock "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/dvc.lock"
    end

    %% CI/CD Layer
    subgraph "CI/CD & Testing"
        direction TB
        CIWorkflow["CI/CD Workflow"]:::cicd
        click CIWorkflow "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/.github/workflows/ci.yaml"
        TestEnv["Test Environment Validation"]:::cicd
        click TestEnv "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/test_environment.py"
        ToxConfig["Test Harness Configuration"]:::cicd
        click ToxConfig "https://github.com/ayushigupta9723/spotify-recommender-system/blob/main/tox.ini"
    end

    %% Dependencies
    ExternalData -->|"ingest"| RawData
    RawData -->|"clean"| DataCleaning
    DataCleaning --> InterimData
    InterimData --> ContentFE
    InterimData --> CollabFE
    ContentFE --> ProcessedData
    CollabFE --> ProcessedData
    ProcessedData --> HybridModel
    HybridModel --> ModelArtifacts
    HybridModel --> TransformerArtifact
    ProcessedData --> Visualization
    ModelArtifacts --> Visualization
    Visualization --> Reports
    Visualization --> Docs
    Visualization --> Notebooks
    TransformerArtifact --> APIApp
    ModelArtifacts --> APIApp

    %% Orchestration Flows
    Makefile -.-> DVCPipeline
    Makefile -.-> DVCLock
    DVCPipeline -.-> DataCleaning
    DVCPipeline -.-> ContentFE
    DVCPipeline -.-> CollabFE
    DVCPipeline -.-> HybridModel
    DVCPipeline -.-> Visualization
    CIWorkflow -.-> DVCPipeline
    CIWorkflow -.-> Makefile
    CIWorkflow -.-> DockerDef

    %% Serving Build
    DockerDef -.-> APIApp

    classDef data fill:#dae8fc,stroke:#6c8ebf,color:#08427b;
    classDef processing fill:#d5e8d4,stroke:#82b366,color:#274e13;
    classDef modeling fill:#f8cecc,stroke:#b85450,color:#7f1d1d;
    classDef reporting fill:#fff2cc,stroke:#d6b656,color:#7a5b1a;
    classDef serving fill:#e1d5e7,stroke:#9673a6,color:#4b1845;
    classDef orchestration fill:#f2f2f2,stroke:#888888,color:#333333;
    classDef cicd fill:#f2f2f2,stroke:#888888,color:#333333;
```
