How bhavik-ksvss/cloud-movie-recommender-system works
Main Components
• Data Ingestion:
– Represent the external dataset source (MovieLens) and the initial data transfer using Cloud Shell commands.
– Show the transition from the external source to a temporary staging area before moving the data.
• Storage:
– Optionally, show the archive directory with raw CSV files as reference to the dataset format.
• Data Processing & Analysis:
– Include the Dataproc cluster as a central component.
– Distinguish the Jupyter Notebook (“Recommendation System.ipynb”) that performs initial data transformation, summary statistics, and visualizations.
• Recommendation Engine:
– Illustrate the PySpark job driven by the “recommender.py” script that runs on the Dataproc cluster, executing the collaborative filtering algorithm.
• Data Warehousing:
– Represent Google BigQuery as the target where the processed data, including predictions, is migrated and stored for querying.
• Additional GCP Components:
– Optionally, add elements for Virtual Private Cloud (VPC) networks and Cloud Shell to capture the complete GCP ecosystem used.
Relationships and Data Flow
• From GCS, show the data flowing into the “Dataproc Cluster” where two parallel activities occur:
– A Jupyter Notebook handling exploratory analysis and visualizations.
– A PySpark job (initiated by recommender.py) that performs collaborative filtering and produces recommendations.
• Both processes feed results into “BigQuery”, indicating that the processed dataset and predictions are migrated into tables.
Architectural Patterns and Technologies
• Separation of concerns: ingestion, transformation/analysis, recommendation engine, and data warehousing.
• Usage of cloud-native services, where different components (GCS, Dataproc, BigQuery) are orchestrated on GCP.
• Mention that the alternating least squares algorithm (a key point of the recommendation engine) is implemented within the Spark job.