How mjngxwnj/big-data-for-music-recommendation-system works
Project’s Main Components
• Data Ingestion & Collection: This includes fetching raw artist data from Kworb.net via web scraping and calling the Spotify API for detailed artist, album, and track information. Also note the initial CSV storage and daily incremental scraping approach.
• Data Storage Layers:
• Raw storage in MongoDB (storing raw API responses and CSV data).
• Data Lake Storage using HDFS to maintain the three-layered processing (Bronze, Silver, Gold).
• Data Processing & Transformation:
• Bronze Layer: Minimal processing of raw data, schema application and incremental load.
• Silver Layer: Data cleaning steps (dropping columns, null handling, renaming) implemented by the SilverLayer class.
• Gold Layer: Schema combining and final data integration for the Snowflake-based Data Warehouse.
• Data Warehouse & Reporting:
• Loading cleansed and integrated data into Snowflake.
• Visualization and reporting via PowerBI.
• ETL Workflow & Orchestration:
• Apache Airflow orchestration is used to schedule and run the scraping, ETL, and data warehouse load tasks. Look for task groups such as branch_task, initial_task, crawl_spotify_data_taskgroup, ETL_HDFS_taskgroup, create_database_schema_task, and warehouse_load_task.
• Machine Learning Recommendation Engine:
• Two distinct approaches are implemented – one with Content-Based Filtering (using CountVectorizer, cosine similarity, etc.) and another using K-means clustering combined with PCA for visualization.
• Application Layer:
• The project includes a front-end and back-end built with Streamlit (as can be seen with streamlit_frontent.py and streamlit_backend.py) to deliver the recommendation system demo.
Relationships and Data Flow
• Flow then into MongoDB for initial storage.
• An ETL pipeline (managed by Airflow) that picks up data from MongoDB and performs three-layer processing on HDFS:
• Bronze → Silver → Gold layers.
• From Gold, data is transferred to the Snowflake Data Warehouse.
• Also, include a separate path for the interactive application (Streamlit) which leverages the recommendation model output.
Architectural Patterns and Design Principles
• The project follows a layered architecture (data lake with Bronze, Silver, Gold layers) and a pipeline pattern for incremental data processing.
• There is a clear separation between data ingestion, processing, storage, and presentation layers.
• Microservice-ish principles are present: individual containerized components (note the various Dockerfiles, docker-compose.yaml) for the application and for the Airflow pipelines ensuring isolated, scalable deployments.
• Use container orchestration as a design principle to represent Docker’s role in creating consistent runtime environments.
Technologies and Tools to Annotate
• Python is the main programming language.
• Docker and Docker Compose are used for containerization.
• Apache Airflow for workflow orchestration.
• MongoDB as the NoSQL raw datastore.
• HDFS for the data lake.
• Apache Spark (with PySpark for schema definitions and processing) is inferred for ETL processing.
• Snowflake as the Final Data Warehouse.
• PowerBI for data visualization.
• Streamlit for the application UI.
• Additionally, include the use of libraries and connectors like the Snowflake JDBC drivers and Spark-Snowflake connector present in the jars directory.