How vaibhav3792/movie-recommender-system works
1. Top-level shape
- This repo is an ML recommender system, not a general web app.
- It has two main phases:
- Offline pipeline: ingest/clean data, engineer movie features, train recommendation models.
- Online inference/UI: a Streamlit app serves recommendations and explanations.
2. Data sources and repository-local data
- The project is built around MovieLens 25M ratings plus TMDB-style movie metadata.
- The only raw mapping file visible in the repo is data/raw/links.csv, which is typically used to connect MovieLens movie IDs to TMDB IDs.
- The cleaned output shown in the tree is data/processed/movies_clean.csv, which is the likely canonical input to the recommendation models.
3. src/data_pipeline.py is the preprocessing boundary
- This module is the data engineering layer.
- Its role is to merge/clean the movie metadata, normalize fields like genres/cast/director/overview, and produce the processed dataset used by downstream models.
- Architecturally, it sits between raw datasets and all model training code.
4. Content-based recommendation subsystem
- src/content_model.py implements the movie-content side of the system.
- From the README, this combines:
- TF-IDF over movie tags/text fields
- GloVe word embeddings for semantic similarity
- Input is enriched movie metadata; output is a content-similarity score or ranked candidate list based on shared themes, actors, directors, and text semantics.
5. Collaborative filtering subsystem
- src/collaborative_model.py handles the rating-based recommender.
- It uses matrix factorization / SVD over the MovieLens ratings matrix to learn latent user/movie preferences.
- This is the only subsystem that depends on the large user-rating graph rather than movie metadata.
6. Hybrid fusion layer
- src/hybrid_model.py is the system’s main recommendation combiner.
- It blends the content score and collaborative score into a final ranking, matching the README’s described weighted ensemble.
- This module is the core architectural bridge between “movie similarity” and “taste prediction.”
7. Explainability layer
- src/explainer.py adds user-facing rationale to recommendations.
- It likely derives reasons from metadata overlap such as:
- similar genres
- shared cast
- same director
- similar themes
- This is a presentation-facing subsystem, but it depends on the same cleaned metadata used by the content model.
8. Serving/UI layer
- app.py is the runtime entrypoint and the app file referenced by the Hugging Face Spaces config.
- It likely loads the trained artifacts, accepts a movie selection or similar input, calls the hybrid recommender, then renders the ranked results and explanations in the UI.
- This is the only interactive boundary in the repo.
9. Model and artifact flow
- The README indicates large model files are not stored in the repo and are instead loaded from the Hugging Face Hub on first run.
- That means the architecture separates:
- source code and lightweight data in the repo
- large trained artifacts in external model storage
- This is important for diagramming because inference depends on remote artifact retrieval.
10. Deployment/runtime
- Dockerfile and the README metadata show the app is deployed as a Docker-based Hugging Face Space.
- The operational runtime is therefore containerized, with app.py as the startup entrypoint.
- requirements.txt defines the Python ML stack needed at runtime: pandas/numpy, scikit-learn/NLTK, Gensim, Surprise, and Streamlit.
11. Exploratory vs production code
- notebooks/development.ipynb is clearly exploratory/development-only.
- The production path is the src/ modules plus app.py; the notebook is not part of the serving architecture.
12. End-to-end data flow
- Raw metadata and ratings → data_pipeline.py → cleaned movie dataset/features.
- Cleaned metadata → content_model.py and explainer.py.
- Ratings matrix → collaborative_model.py.
- Both scoring paths → hybrid_model.py.
- Final ranked recommendations + reasons → app.py UI.
13. Key architectural boundary to draw
- The most important boundary is between offline ML preparation/training and online recommendation serving.
- Another important boundary is between metadata-based similarity and ratings-based latent preference modeling, with hybrid_model.py merging them into a single user-facing result.
Main entry points: data_pipeline.py Data Pipeline · __init__.py Src Package · app.py App · hybrid_model.py Hybrid Model
Components 16
Offline ML
Data preparation and model-building components.
- Data Pipeline · Cleans and enriches movie metadata into the processed dataset used downstream.
src/data_pipeline.py - Content Model · Computes movie similarity from text and metadata signals such as tags, genres, cast, and embeddings.
src/content_model.py - Collaborative Model · Learns latent preferences from the user-item ratings matrix using collaborative filtering / SVD.
src/collaborative_model.py - Hybrid Model · Combines content-based and collaborative scores into the final recommendation ranking.
src/hybrid_model.py - Notebook · Development-only exploratory notebook, outside the production serving path.
notebooks/development.ipynb - Src Package · Package marker for the model and pipeline modules.
src/__init__.py
Online Serving
Runtime inference and user-facing recommendation delivery.
Data Assets
Raw and processed repository-local datasets.
- Cleaned Movies · Canonical cleaned movie metadata consumed by the recommendation models.
data/processed/movies_clean.csv - ID Links · Maps MovieLens movie identifiers to TMDB-style identifiers used in metadata joins.
data/raw/links.csv
External Runtime
Container and remote artifact dependencies.
- Model Hub · External source for large trained artifacts loaded on first run rather than stored in the repo.
- Ratings Data · Large MovieLens ratings corpus that drives collaborative filtering.
- Movie Metadata · TMDB-style movie metadata source used to enrich and clean movie features.
- Docker · Containerized deployment environment for the Streamlit application.
Dockerfile - Python Stack · Runtime ML and UI dependency set used by the app and models.
requirements.txt - README · Repository overview and setup instructions describing the pipeline and deployment flow.
README.md
Connections 24
- ID Links → Data Pipeline: ID mapping · Raw movie ID links support metadata joins during preprocessing.
- Movie Metadata → Data Pipeline: enrich · External movie metadata is cleaned and normalized into the processed dataset.
- Data Pipeline → Cleaned Movies: writes · Preprocessing produces the canonical cleaned movie table.
- Cleaned Movies → Content Model: features · Cleaned metadata feeds text and metadata-based similarity modeling.
- Cleaned Movies → Explainer: evidence · Clean metadata provides the fields used to explain recommendations.
- Ratings Data → Collaborative Model: train · Ratings data is used to fit collaborative filtering factors.
- Content Model → Hybrid Model: content score · Content-based ranking contributes one side of the ensemble.
- Collaborative Model → Hybrid Model: collab score · Latent preference scores contribute the other side of the ensemble.
- Hybrid Model → App: ranked recs · The fused recommender returns the final ranked list to the UI.
- Explainer → App: reasons · Explanations are rendered alongside recommendations.
- Model Hub → App: load artifacts · The runtime fetches large trained artifacts externally on first use.
- Python Stack → App: deps · Python dependencies support the serving runtime and model libraries.
- Docker → App: start · Container startup launches the Streamlit application entrypoint.
- README → Data Pipeline: describes · Project documentation outlines the offline data preparation flow.
- README → Hybrid Model: describes · Documentation explains the hybrid ensemble architecture.
- README → App: describes · Documentation covers the deployed app and usage path.
- Notebook → Data Pipeline: prototypes · Exploratory analysis informs preprocessing and modeling decisions.
- Notebook → Content Model: prototypes · Notebook experimentation supports the content recommender design.
- Notebook → Collaborative Model: prototypes · Notebook experimentation supports collaborative filtering development.
- Src Package → Data Pipeline: package · Marks the source directory as a Python package for the model modules.
- Src Package → Content Model: package · Marks the source directory as a Python package for the model modules.
- Src Package → Collaborative Model: package · Marks the source directory as a Python package for the model modules.
- Src Package → Hybrid Model: package · Marks the source directory as a Python package for the model modules.
- Src Package → Explainer: package · Marks the source directory as a Python package for the model modules.