# ayushd64/netflix-behavior-transformer-recommendation-engine architecture

> Architecture diagram of the GitHub repository ayushd64/netflix-behavior-transformer-recommendation-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushd64/netflix-behavior-transformer-recommendation-engine
- Repository: https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine

## Overview

1. **What this repo is**
   This is a MovieLens-100k recommendation system organized around an offline training/indexing pipeline and a live inference service. The core runtime path is a two-stage recommender: a **sequence-based retriever** followed by a **neural ranker**.

2. **Data layer**
   The raw dataset lives under `movielens_pipeline/data/ml-100k/`, including ratings, user/item metadata, and the standard train/test splits (`u1.base`, `u1.test`, etc.). These files are the source of truth for both model training and evaluation.

3. **Offline model training**
   `movielens_pipeline/train_sequence_transformer.py` trains the sequential behavior model described in the README, producing `sequential_transformer_retrieval.pth`.  
   `movielens_pipeline/pytorch_movielens.py` appears to train the optimized collaborative-filtering model, producing `optimized_ncf_model.pth`.

4. **Retrieval index construction**
   `movielens_pipeline/build_realtime_index.py` converts learned item representations into a Faiss index stored as `realtime_movies.index`. This is the retrieval layer used at serving time, so the online system does not need to scan the full catalog.

5. **Online serving path**
   `movielens_pipeline/streaming_api.py` is the main runtime entry point. The README shows it running under FastAPI/Uvicorn and exposing a `/watch/` endpoint. The live request includes a `user_id` and a watched title, updates session state, retrieves candidates, and returns ranked recommendations.

6. **Two-stage recommendation flow**
   The request path is: incoming watch event → sequence/behavior encoder creates a session representation → Faiss returns a small candidate set → the NCF model scores those candidates → results are sorted and returned as JSON. The architecture boundary is clear: **retrieval** is optimized for recall and speed, while **ranking** is optimized for precision.

7. **State and mapping artifacts**
   `movielens_pipeline/id_mappings.json` and `movielens_pipeline/realtime_mappings.json` hold the lookup tables needed to translate between MovieLens identifiers, internal model IDs, and index positions. These artifacts are essential glue between the dataset, the trained models, and the vector index.

8. **Evaluation and diagnostics**
   `movielens_pipeline/benchmark_pipeline.py` measures end-to-end latency/throughput for the live pipeline. `movielens_pipeline/visualize_faiss.py` is a diagnostic tool for inspecting the vector space, likely via dimensionality reduction plots.

9. **Exploratory and baseline recommenders**
   The root-level scripts `content_based.py`, `matrix_factorization.py`, `hybrid_recommendation.py`, `neural_recommendation.py`, `scratch_recommendation.py`, and `vector_indexing.py` look like earlier iterations or alternative approaches. They form a prototyping/baseline layer rather than the main serving architecture.

10. **Primary architectural boundaries**
   The repository cleanly separates:
   - **data ingestion/prep** from `data/ml-100k/`
   - **model training** (`train_sequence_transformer.py`, `pytorch_movielens.py`)
   - **index building** (`build_realtime_index.py`)
   - **online inference** (`streaming_api.py`, possibly `recommendation_api.py`)
   - **benchmarking/visualization** (`benchmark_pipeline.py`, `visualize_faiss.py`)

11. **Deployment/runtime dependencies that matter**
   The material runtime stack is **PyTorch** for model inference/training, **Faiss** for approximate nearest-neighbor retrieval, and **FastAPI + Uvicorn** for the serving layer. These directly shape the architecture: GPU/CPU model inference, in-memory session state, and low-latency vector search.

12. **Likely diagram to draw**
   A useful diagram would show offline artifacts feeding online serving: raw MovieLens data → training scripts → `.pth` model weights + mapping JSON + Faiss index → FastAPI streaming service → candidate retrieval → NCF ranking → response. That is the central architecture of the repo.

Main entry points: [`movielens_pipeline/streaming_api.py`](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/streaming_api.py) (Streaming API), [`movielens_pipeline/data/ml-100k`](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/tree/main/movielens_pipeline/data/ml-100k) (MovieLens 100k), [`movielens_pipeline/train_sequence_transformer.py`](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/train_sequence_transformer.py) (Seq trainer), [`movielens_pipeline/build_realtime_index.py`](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/build_realtime_index.py) (Index builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_data["Data"]
  node_n_data[("MovieLens 100k<br/>dataset")]
end

subgraph group_g_offline["Offline pipeline"]
  node_n_train_seq["Seq trainer<br/>training script"]
  node_n_train_ncf["NCF trainer<br/>training script"]
  node_n_build_index["Index builder"]
  node_n_benchmark["Benchmarking<br/>diagnostic"]
  node_n_visualize["Faiss viz<br/>diagnostic<br/>[visualize_faiss.py]"]
  node_n_seq_weights["Seq weights<br/>model artifact"]
  node_n_ncf_weights["NCF weights<br/>model artifact"]
end

subgraph group_g_online["Online service"]
  node_n_index[("Faiss index<br/>index artifact")]
  node_n_idmap["ID map<br/>mapping artifact<br/>[id_mappings.json]"]
  node_n_rtmap["Realtime map<br/>mapping artifact"]
  node_n_stream["Streaming API<br/>serving app<br/>[streaming_api.py]"]
  node_n_api["Rec API<br/>serving app"]
  node_n_retrieval{{"Retriever<br/>runtime stage"}}
  node_n_ranking{{"Ranker<br/>runtime stage"}}
end

subgraph group_g_proto["Prototypes"]
  node_n_content["Content based<br/>prototype<br/>[content_based.py]"]
  node_n_matrix["Matrix factorization<br/>prototype"]
  node_n_hybrid["Hybrid rec<br/>prototype"]
  node_n_neural["Neural rec<br/>prototype"]
  node_n_scratch["Scratch rec<br/>prototype"]
  node_n_vector["Vector indexing<br/>prototype<br/>[vector_indexing.py]"]
end

node_n_data -->|"training data"| node_n_train_seq
node_n_data -->|"training data"| node_n_train_ncf
node_n_data -->|"catalog"| node_n_build_index
node_n_train_seq -->|"produces"| node_n_seq_weights
node_n_train_ncf -->|"produces"| node_n_ncf_weights
node_n_train_seq -->|"embeddings"| node_n_build_index
node_n_build_index -->|"writes"| node_n_index
node_n_data -->|"defines"| node_n_idmap
node_n_data -->|"defines"| node_n_rtmap
node_n_seq_weights -->|"loads"| node_n_stream
node_n_ncf_weights -->|"loads"| node_n_stream
node_n_index -->|"candidate source"| node_n_retrieval
node_n_idmap -->|"lookup"| node_n_stream
node_n_rtmap -->|"lookup"| node_n_stream
node_n_stream -->|"request flow"| node_n_retrieval
node_n_retrieval -->|"candidates"| node_n_ranking
node_n_ranking -->|"response"| node_n_stream
node_n_benchmark -.->|"measures"| node_n_stream
node_n_visualize -.->|"inspects"| node_n_index
node_n_api -.->|"alternate entry"| node_n_stream
node_n_content -.->|"baseline"| node_n_hybrid
node_n_matrix -.->|"baseline"| node_n_hybrid
node_n_neural -.->|"baseline"| node_n_hybrid
node_n_scratch -.->|"experiments"| node_n_vector

click node_n_data "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/tree/main/movielens_pipeline/data/ml-100k"
click node_n_train_seq "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/train_sequence_transformer.py"
click node_n_train_ncf "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/pytorch_movielens.py"
click node_n_build_index "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/build_realtime_index.py"
click node_n_benchmark "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/benchmark_pipeline.py"
click node_n_visualize "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/visualize_faiss.py"
click node_n_seq_weights "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/sequential_transformer_retrieval.pth"
click node_n_ncf_weights "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/optimized_ncf_model.pth"
click node_n_index "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/realtime_movies.index"
click node_n_idmap "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/id_mappings.json"
click node_n_rtmap "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/realtime_mappings.json"
click node_n_stream "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/streaming_api.py"
click node_n_api "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/recommendation_api.py"
click node_n_content "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/content_based.py"
click node_n_matrix "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/matrix_factorization.py"
click node_n_hybrid "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/hybrid_recommendation.py"
click node_n_neural "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/neural_recommendation.py"
click node_n_scratch "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/scratch_recommendation.py"
click node_n_vector "https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/vector_indexing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_data toneBlue
class node_n_train_seq,node_n_train_ncf,node_n_build_index,node_n_benchmark,node_n_visualize,node_n_seq_weights,node_n_ncf_weights toneAmber
class node_n_index,node_n_idmap,node_n_rtmap,node_n_stream,node_n_api,node_n_retrieval,node_n_ranking toneMint
class node_n_content,node_n_matrix,node_n_hybrid,node_n_neural,node_n_scratch,node_n_vector toneRose
```

## Components

### Data

Raw MovieLens files and split artifacts used for training and evaluation.

- [MovieLens 100k](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/tree/main/movielens_pipeline/data/ml-100k): dataset · Raw ratings, user/item metadata, and train/test splits. · `movielens_pipeline/data/ml-100k`

### Offline pipeline

Scripts that train models, build retrieval artifacts, and benchmark the system.

- [Seq trainer](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/train_sequence_transformer.py): training script · Trains the sequence-based transformer retriever. · `movielens_pipeline/train_sequence_transformer.py`
- [NCF trainer](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/pytorch_movielens.py): training script · Trains the optimized neural collaborative filtering ranker. · `movielens_pipeline/pytorch_movielens.py`
- [Index builder](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/build_realtime_index.py): index build · Converts learned item vectors into a Faiss retrieval index. · `movielens_pipeline/build_realtime_index.py`
- [Benchmarking](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/benchmark_pipeline.py): diagnostic · Measures latency and throughput of the live pipeline. · `movielens_pipeline/benchmark_pipeline.py`
- [Faiss viz](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/visualize_faiss.py): diagnostic · Explores the vector space and retrieval structure. · `movielens_pipeline/visualize_faiss.py`
- [Seq weights](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/sequential_transformer_retrieval.pth): model artifact · Saved sequential transformer retrieval weights. · `movielens_pipeline/sequential_transformer_retrieval.pth`
- [NCF weights](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/optimized_ncf_model.pth): model artifact · Saved neural collaborative filtering model weights. · `movielens_pipeline/optimized_ncf_model.pth`

### Online service

Live request handling, retrieval, and ranking path.

- [Faiss index](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/realtime_movies.index): index artifact · Prebuilt nearest-neighbor index used for candidate retrieval. · `movielens_pipeline/realtime_movies.index`
- [ID map](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/id_mappings.json): mapping artifact · Translates between MovieLens IDs and internal model identifiers. · `movielens_pipeline/id_mappings.json`
- [Realtime map](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/realtime_mappings.json): mapping artifact · Maps live-serving identifiers to index positions and model IDs. · `movielens_pipeline/realtime_mappings.json`
- [Streaming API](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/streaming_api.py): serving app · FastAPI/Uvicorn runtime entry point for watch events and recommendations. · `movielens_pipeline/streaming_api.py`
- [Rec API](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/movielens_pipeline/recommendation_api.py): serving app · Alternative API entry point for recommendations. · `movielens_pipeline/recommendation_api.py`
- **Retriever**: runtime stage · Uses the session representation to fetch a small candidate set.
- **Ranker**: runtime stage · Scores retrieved candidates and sorts the final recommendations.

### Prototypes

Alternative recommendation approaches and exploratory scripts.

- [Content based](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/content_based.py): prototype · Standalone baseline recommender implementation. · `content_based.py`
- [Matrix factorization](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/matrix_factorization.py): prototype · Collaborative-filtering baseline implementation. · `matrix_factorization.py`
- [Hybrid rec](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/hybrid_recommendation.py): prototype · Mixed recommender prototype combining multiple signals. · `hybrid_recommendation.py`
- [Neural rec](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/neural_recommendation.py): prototype · Earlier neural recommendation experiment. · `neural_recommendation.py`
- [Scratch rec](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/scratch_recommendation.py): prototype · Experimental scratch-built recommender. · `scratch_recommendation.py`
- [Vector indexing](https://github.com/ayushd64/netflix-behavior-transformer-recommendation-engine/blob/main/vector_indexing.py): prototype · Prototype for vector indexing and retrieval experiments. · `vector_indexing.py`

## Connections

- MovieLens 100k → Seq trainer: training data · MovieLens interactions and metadata feed sequence model training.
- MovieLens 100k → NCF trainer: training data · MovieLens data feeds NCF training.
- MovieLens 100k → Index builder: catalog · Item metadata and learned representations are used to build the retrieval index.
- Seq trainer → Seq weights: produces · Trained sequence model weights are saved for serving.
- NCF trainer → NCF weights: produces · Trained ranker weights are saved for online scoring.
- Seq trainer → Index builder: embeddings · Sequence model outputs item vectors used for indexing.
- Index builder → Faiss index: writes · Build step persists the Faiss index artifact.
- MovieLens 100k → ID map: defines · Dataset identifiers are normalized into internal mappings.
- MovieLens 100k → Realtime map: defines · Serving-time identifier mappings are derived from the dataset.
- Seq weights → Streaming API: loads · Streaming service loads the retriever model.
- NCF weights → Streaming API: loads · Streaming service loads the ranker model.
- Faiss index → Retriever: candidate source · Retriever queries the prebuilt Faiss index.
- ID map → Streaming API: lookup · Service resolves external user and item identifiers.
- Realtime map → Streaming API: lookup · Service uses realtime mappings for index and model IDs.
- Streaming API → Retriever: request flow · Incoming watch events are transformed into retrieval queries.
- Retriever → Ranker: candidates · Retriever passes a small candidate set to the ranker.
- Ranker → Streaming API: response · Ranked recommendations are returned as JSON.
- Benchmarking → Streaming API: measures · Benchmarking targets the live service.
- Faiss viz → Faiss index: inspects · Visualization reads the retrieval index for diagnostics.
- Rec API → Streaming API: alternate entry · Secondary API surface likely reuses the same serving stack.
- Content based → Hybrid rec: baseline · Alternative recommender experiments inform hybrid approaches.
- Matrix factorization → Hybrid rec: baseline · Matrix factorization is one of the signals used in hybrid work.
- Neural rec → Hybrid rec: baseline · Neural recommender experiments connect to the hybrid line.
- Scratch rec → Vector indexing: experiments · Prototype scripts share exploratory vector indexing ideas.