How ayushigupta9723/spotify-recommender-system works
Project Type and Purpose
• Type: A data-science / ML pipeline for music recommendation (content-based, collaborative, hybrid).
• Purpose: Ingest raw music and user data, clean & transform it, engineer features, train/evaluate models, and deploy an inference API (via app.py).
Main Components
a. Data Sources & Storage
– “external” data store (3rd-party CSV)
– “raw” data folder (immutable dumps under DVC control)
– “interim” (transformed but not final)
– “processed” (final datasets ready for modeling)
– DVC remote (could be S3/GCS for versioned artifacts)
b. Pipeline Orchestrator
– DVC (dvc.yaml, dvc.lock) drives the stages: data download → cleaning → feature extraction → model training → evaluation → export.
– Makefile front-ends common commands (make data, make train, etc.).
c. Data-Processing & Feature Engineering Modules
– src/data/data_cleaning.py
– src/features/content_based_filtering.py
– src/features/collabrative_filtering.py
d. Model Training & Inference
– src/models/hybrid_recommendation.py (combines content + collaborative)
– src/models/transform.py (data transforms)
– transformer.joblib (saved preprocessing pipeline)
– models/ (folder for serialized models, predictions)
e. Visualization & Reporting
– reports/figures and docs/ (Sphinx docs + notebooks/ for EDA)
f. Deployment & API
– app.py (Flask/FastAPI script exposing endpoints for recommendations)
– Dockerfile (containerizes app + pipeline environment)
g. CI/CD & Testing
– .github/workflows/ci.yaml (runs tests, lint, dvc repro checks)
– test_environment.py, tox.ini (test harness, environment validation)
Relationships & Data Flow
1. Raw CSV data (data/external, data/raw) → DVC pulls it in
2. data_cleaning.py → writes to data/interim
3. feature scripts → consume interim → produce data/processed or feature artifacts
4. model training (hybrid_recommendation) → reads features → outputs model artifacts into models/ + transformer.joblib
5. Visualization scripts & Jupyter notebooks read processed data + model outputs → generate reports & figures
6. app.py loads transformer.joblib + model files → serves online recommendations
7. CI pipeline validates code, runs dvc repro, tests, builds Docker image, and can push to registry
Architectural Patterns & Principles
– Modular pipeline stages (single responsibility per script folder)
– Versioned data + artifacts via DVC for reproducibility
– Clear separation: Data layer, Processing/Modeling layer, Serving layer, CI/CD
– Containerization for environment consistency (Docker)
– Infrastructure-as-code style orchestration (Make + DVC + GitHub Actions)
– Rectangles for code modules (data cleaning, feature eng, modeling, visualization, app)
– GitHub logo for CI/CD workflow
– Data Layer (all data folders + DVC remote)
– Processing Layer (src/data, src/features)
– Modeling Layer (src/models + models/)
– Serving Layer (app.py, Dockerfile)
– Reporting Layer (notebooks, src/visualization, docs, reports)
– Orchestration Layer (Makefile, dvc.yaml, GitHub Actions)
– Label every data store and pipeline stage clearly.
– Don’t forget external dependencies: GitHub Actions, Docker registry, any external data sources or APIs.