How ayushd64/netflix-behavior-transformer-recommendation-engine works
1. What this repo is
This is a MovieLens-100k recommendation system organized around an offline training/indexing pipeline and a live inference service. The core runtime path is a two-stage recommender: a sequence-based retriever followed by a neural ranker.
2. Data layer
The raw dataset lives under movielens_pipeline/data/ml-100k/, including ratings, user/item metadata, and the standard train/test splits (u1.base, u1.test, etc.). These files are the source of truth for both model training and evaluation.
3. Offline model training
movielens_pipeline/train_sequence_transformer.py trains the sequential behavior model described in the README, producing sequential_transformer_retrieval.pth.
movielens_pipeline/pytorch_movielens.py appears to train the optimized collaborative-filtering model, producing optimized_ncf_model.pth.
4. Retrieval index construction
movielens_pipeline/build_realtime_index.py converts learned item representations into a Faiss index stored as realtime_movies.index. This is the retrieval layer used at serving time, so the online system does not need to scan the full catalog.
5. Online serving path
movielens_pipeline/streaming_api.py is the main runtime entry point. The README shows it running under FastAPI/Uvicorn and exposing a /watch/ endpoint. The live request includes a user_id and a watched title, updates session state, retrieves candidates, and returns ranked recommendations.
6. Two-stage recommendation flow
The request path is: incoming watch event → sequence/behavior encoder creates a session representation → Faiss returns a small candidate set → the NCF model scores those candidates → results are sorted and returned as JSON. The architecture boundary is clear: retrieval is optimized for recall and speed, while ranking is optimized for precision.
7. State and mapping artifacts
movielens_pipeline/id_mappings.json and movielens_pipeline/realtime_mappings.json hold the lookup tables needed to translate between MovieLens identifiers, internal model IDs, and index positions. These artifacts are essential glue between the dataset, the trained models, and the vector index.
8. Evaluation and diagnostics
movielens_pipeline/benchmark_pipeline.py measures end-to-end latency/throughput for the live pipeline. movielens_pipeline/visualize_faiss.py is a diagnostic tool for inspecting the vector space, likely via dimensionality reduction plots.
9. Exploratory and baseline recommenders
The root-level scripts content_based.py, matrix_factorization.py, hybrid_recommendation.py, neural_recommendation.py, scratch_recommendation.py, and vector_indexing.py look like earlier iterations or alternative approaches. They form a prototyping/baseline layer rather than the main serving architecture.
10. Primary architectural boundaries
The repository cleanly separates:
- data ingestion/prep from data/ml-100k/
- model training (train_sequence_transformer.py, pytorch_movielens.py)
- index building (build_realtime_index.py)
- online inference (streaming_api.py, possibly recommendation_api.py)
- benchmarking/visualization (benchmark_pipeline.py, visualize_faiss.py)
11. Deployment/runtime dependencies that matter
The material runtime stack is PyTorch for model inference/training, Faiss for approximate nearest-neighbor retrieval, and FastAPI + Uvicorn for the serving layer. These directly shape the architecture: GPU/CPU model inference, in-memory session state, and low-latency vector search.
12. Likely diagram to draw
A useful diagram would show offline artifacts feeding online serving: raw MovieLens data → training scripts → .pth model weights + mapping JSON + Faiss index → FastAPI streaming service → candidate retrieval → NCF ranking → response. That is the central architecture of the repo.
Main entry points: streaming_api.py Streaming API · ml-100k MovieLens 100k · train_sequence_transformer.py Seq trainer · build_realtime_index.py Index builder
Components 21
Data
Raw MovieLens files and split artifacts used for training and evaluation.
- MovieLens 100k · Raw ratings, user/item metadata, and train/test splits.
movielens_pipeline/data/ml-100k
Offline pipeline
Scripts that train models, build retrieval artifacts, and benchmark the system.
- Seq trainer · Trains the sequence-based transformer retriever.
movielens_pipeline/train_sequence_transformer.py - NCF trainer · Trains the optimized neural collaborative filtering ranker.
movielens_pipeline/pytorch_movielens.py - Index builder · Converts learned item vectors into a Faiss retrieval index.
movielens_pipeline/build_realtime_index.py - Benchmarking · Measures latency and throughput of the live pipeline.
movielens_pipeline/benchmark_pipeline.py - Faiss viz · Explores the vector space and retrieval structure.
movielens_pipeline/visualize_faiss.py - Seq weights · Saved sequential transformer retrieval weights.
movielens_pipeline/sequential_transformer_retrieval.pth - NCF weights · Saved neural collaborative filtering model weights.
movielens_pipeline/optimized_ncf_model.pth
Online service
Live request handling, retrieval, and ranking path.
- Faiss index · Prebuilt nearest-neighbor index used for candidate retrieval.
movielens_pipeline/realtime_movies.index - ID map · Translates between MovieLens IDs and internal model identifiers.
movielens_pipeline/id_mappings.json - Realtime map · Maps live-serving identifiers to index positions and model IDs.
movielens_pipeline/realtime_mappings.json - Streaming API · FastAPI/Uvicorn runtime entry point for watch events and recommendations.
movielens_pipeline/streaming_api.py - Rec API · Alternative API entry point for recommendations.
movielens_pipeline/recommendation_api.py - Retriever · Uses the session representation to fetch a small candidate set.
- Ranker · Scores retrieved candidates and sorts the final recommendations.
Prototypes
Alternative recommendation approaches and exploratory scripts.
- Content based · Standalone baseline recommender implementation.
content_based.py - Matrix factorization · Collaborative-filtering baseline implementation.
matrix_factorization.py - Hybrid rec · Mixed recommender prototype combining multiple signals.
hybrid_recommendation.py - Neural rec · Earlier neural recommendation experiment.
neural_recommendation.py - Scratch rec · Experimental scratch-built recommender.
scratch_recommendation.py - Vector indexing · Prototype for vector indexing and retrieval experiments.
vector_indexing.py
Connections 24
- MovieLens 100k → Seq trainer: training data · MovieLens interactions and metadata feed sequence model training.
- MovieLens 100k → NCF trainer: training data · MovieLens data feeds NCF training.
- MovieLens 100k → Index builder: catalog · Item metadata and learned representations are used to build the retrieval index.
- Seq trainer → Seq weights: produces · Trained sequence model weights are saved for serving.
- NCF trainer → NCF weights: produces · Trained ranker weights are saved for online scoring.
- Seq trainer → Index builder: embeddings · Sequence model outputs item vectors used for indexing.
- Index builder → Faiss index: writes · Build step persists the Faiss index artifact.
- MovieLens 100k → ID map: defines · Dataset identifiers are normalized into internal mappings.
- MovieLens 100k → Realtime map: defines · Serving-time identifier mappings are derived from the dataset.
- Seq weights → Streaming API: loads · Streaming service loads the retriever model.
- NCF weights → Streaming API: loads · Streaming service loads the ranker model.
- Faiss index → Retriever: candidate source · Retriever queries the prebuilt Faiss index.
- ID map → Streaming API: lookup · Service resolves external user and item identifiers.
- Realtime map → Streaming API: lookup · Service uses realtime mappings for index and model IDs.
- Streaming API → Retriever: request flow · Incoming watch events are transformed into retrieval queries.
- Retriever → Ranker: candidates · Retriever passes a small candidate set to the ranker.
- Ranker → Streaming API: response · Ranked recommendations are returned as JSON.
- Benchmarking → Streaming API: measures · Benchmarking targets the live service.
- Faiss viz → Faiss index: inspects · Visualization reads the retrieval index for diagnostics.
- Rec API → Streaming API: alternate entry · Secondary API surface likely reuses the same serving stack.
- Content based → Hybrid rec: baseline · Alternative recommender experiments inform hybrid approaches.
- Matrix factorization → Hybrid rec: baseline · Matrix factorization is one of the signals used in hybrid work.
- Neural rec → Hybrid rec: baseline · Neural recommender experiments connect to the hybrid line.
- Scratch rec → Vector indexing: experiments · Prototype scripts share exploratory vector indexing ideas.