# mjngxwnj/big-data-for-music-recommendation-system architecture

> Architecture diagram of the GitHub repository mjngxwnj/big-data-for-music-recommendation-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mjngxwnj/big-data-for-music-recommendation-system
- Repository: https://github.com/mjngxwnj/big-data-for-music-recommendation-system

## Overview

##### Project’s Main Components
- Data Ingestion & Collection: This includes fetching raw artist data from Kworb.net via web scraping and calling the Spotify API for detailed artist, album, and track information. Also note the initial CSV storage and daily incremental scraping approach.
- Data Storage Layers:
- Raw storage in MongoDB (storing raw API responses and CSV data).
- Data Lake Storage using HDFS to maintain the three-layered processing (Bronze, Silver, Gold).
- Data Processing & Transformation:
- Bronze Layer: Minimal processing of raw data, schema application and incremental load.
- Silver Layer: Data cleaning steps (dropping columns, null handling, renaming) implemented by the SilverLayer class.
- Gold Layer: Schema combining and final data integration for the Snowflake-based Data Warehouse.
- Data Warehouse & Reporting:
- Loading cleansed and integrated data into Snowflake.
- Visualization and reporting via PowerBI.
- ETL Workflow & Orchestration:
- Apache Airflow orchestration is used to schedule and run the scraping, ETL, and data warehouse load tasks. Look for task groups such as branch_task, initial_task, crawl_spotify_data_taskgroup, ETL_HDFS_taskgroup, create_database_schema_task, and warehouse_load_task.
- Machine Learning Recommendation Engine:
- Two distinct approaches are implemented – one with Content-Based Filtering (using CountVectorizer, cosine similarity, etc.) and another using K-means clustering combined with PCA for visualization.
- Application Layer:
- The project includes a front-end and back-end built with Streamlit (as can be seen with streamlit_frontent.py and streamlit_backend.py) to deliver the recommendation system demo.

##### Relationships and Data Flow
- Flow then into MongoDB for initial storage.
- An ETL pipeline (managed by Airflow) that picks up data from MongoDB and performs three-layer processing on HDFS:
- Bronze → Silver → Gold layers.
- From Gold, data is transferred to the Snowflake Data Warehouse.
- Also, include a separate path for the interactive application (Streamlit) which leverages the recommendation model output.

##### Architectural Patterns and Design Principles
- The project follows a layered architecture (data lake with Bronze, Silver, Gold layers) and a pipeline pattern for incremental data processing.
- There is a clear separation between data ingestion, processing, storage, and presentation layers.
- Microservice-ish principles are present: individual containerized components (note the various Dockerfiles, docker-compose.yaml) for the application and for the Airflow pipelines ensuring isolated, scalable deployments.
- Use container orchestration as a design principle to represent Docker’s role in creating consistent runtime environments.

##### Technologies and Tools to Annotate
- Python is the main programming language.
- Docker and Docker Compose are used for containerization.
- Apache Airflow for workflow orchestration.
- MongoDB as the NoSQL raw datastore.
- HDFS for the data lake.
- Apache Spark (with PySpark for schema definitions and processing) is inferred for ETL processing.
- Snowflake as the Final Data Warehouse.
- PowerBI for data visualization.
- Streamlit for the application UI.
- Additionally, include the use of libraries and connectors like the Snowflake JDBC drivers and Spark-Snowflake connector present in the jars directory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        ext1["Kworb.net"]:::external
        ext2["Spotify API"]:::external
    end

    %% Data Ingestion Layer
    ingestion["Data Ingestion & Collection"]:::ingestion
    click ingestion "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/application/rcm_music_app.py"
    
    %% MongoDB Raw Storage
    mongodb["MongoDB (Raw Data Storage)"]:::storage
    click mongodb "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/tree/main/pipelines/dags/mongoDB"

    %% Airflow Orchestration
    airflow["Airflow Orchestration"]:::processing
    click airflow "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/mainpipeline.py"
    click airflow "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/init_load.py"
    
    %% ETL Pipeline Subgraph
    subgraph "ETL Pipeline"
        bronze["Bronze Layer Processing"]:::processing
        click bronze "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/spark_script/bronze_script.py"
        
        silver["Silver Layer Processing"]:::processing
        click silver "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/spark_script/silver_script.py"
        click silver "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/spark_script/silver_class.py"

        gold["Gold Layer Processing"]:::processing
        click gold "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/spark_script/gold_script.py"
        
        warehouseLoad["Data Warehouse Load"]:::processing
        click warehouseLoad "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/spark_script/warehouse_load_script.py"
        click warehouseLoad "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/pipelines/dags/sql/create_db_scm_table.py"
    end

    %% Data Warehouse (Snowflake)
    snowflake["Snowflake Data Warehouse"]:::warehouse

    %% Reporting and ML Nodes
    reporting["PowerBI Reporting & Dashboards"]:::reporting
    ml["Machine Learning Recommendation Engine"]:::reporting
    click ml "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/notebook/recommendation_model.ipynb"

    %% Streamlit Application Subgraph
    subgraph "Streamlit Application"
        front["Streamlit Front-End"]:::app
        click front "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/application/streamlit_frontent.py"
        backend["Streamlit Back-End"]:::app
        click backend "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/application/streamlit_backend.py"
    end

    %% Containerization Node
    docker["Docker Containerization"]:::docker
    click docker "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/blob/main/docker-compose.yaml"
    click docker "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/tree/main/application/dockerfile"
    click docker "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/tree/main/pipelines/dockerfile"

    %% JAR Connectors Node
    jar["JAR Connectors"]:::connector
    click jar "https://github.com/mjngxwnj/big-data-for-music-recommendation-system/tree/main/jars/"

    %% Connections
    ext1 -->|"sends Raw Data"| ingestion
    ext2 -->|"sends Raw Data"| ingestion
    ingestion -->|"Extracts Data"| mongodb
    mongodb -->|"Transfers Data"| airflow
    airflow -->|"Triggers"| bronze
    bronze -->|"IncrementalLoad"| silver
    silver -->|"DataCleaning"| gold
    gold -->|"SchemaIntegration"| warehouseLoad
    warehouseLoad -->|"Loads Data"| snowflake
    snowflake -->|"ProvidesData"| reporting
    snowflake -->|"ProvidesDemoData"| front
    snowflake -->|"ProvidesDemoData"| backend
    ml -->|"DeliversRecs"| front

    %% Docker integration (dashed connections)
    ingestion ---|"Containerized"| docker
    airflow ---|"Containerized"| docker
    front ---|"Containerized"| docker
    backend ---|"Containerized"| docker

    %% JAR Connectors integration (dashed connection)
    jar ---|"EnablesConnectivity"| snowflake

    %% Styling
    classDef external fill:#f08080,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef storage fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef processing fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef warehouse fill:#ffa500,stroke:#333,stroke-width:2px;
    classDef reporting fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef app fill:#FFFF99,stroke:#333,stroke-width:2px;
    classDef docker fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef connector fill:#ffcccb,stroke:#333,stroke-width:2px;
```
