yash2010/recommendation_system

Purpose and Entry Points

This is a semantic movie recommendation service: it ingests TMDB metadata, stores movies and user interactions in SQLite, builds a sentence-embedding index, and serves search/recommendation APIs plus a static UI. Runtime entry point is api.py; offline operational entry points are scripts/fetch_tmdb.py, scripts/csv2db.py, scripts/build_index.py, and expander_train.py.

Data Acquisition and Preparation

The TMDB pipeline fetches multilingual movie records, applies filtering/normalization, and exports checkpointed datasets for database import. TMDB is an external dependency authenticated by TMDB_TOKEN.

• data_pipeline/pipeline.py

• data_pipeline/fetcher.py

• data_pipeline/processor.py

• data_pipeline/exporter.py

• data_pipeline/config.yaml

The intended boundary is file-based: pipeline output (CSV or Parquet) is imported into SQLite separately rather than queried directly by the serving layer.

Persistent Catalog and Feedback Store

SQLite is the system of record for movie metadata, users, and interactions. The database layer supports feedback/history needed for eventual personalization, while the current retrieval path remains content-based.

• database.py

• scripts/csv2db.py

User interaction writes (click, rate, watch) enter through the API and are persisted independently of embedding-index generation.

Embedding Index and Retrieval

Index construction embeds each movie using the external Sentence Transformers model all-MiniLM-L6-v2, producing artifact files aligned with movie metadata. At serving time, the recommender embeds the query and performs cosine-similarity ranking over the loaded NumPy embedding matrix; it also supports title lookup, genre filtering, and item-to-item similarity.

• scripts/build_index.py

• recommender.py

• api.py

This creates a deployment consistency boundary: SQLite catalog contents and artifacts/embeddings.npy must be rebuilt/kept aligned after catalog changes.

Query Expansion Subsystem

Before semantic retrieval, requests may be enriched through a pluggable expander abstraction. Ollama is an external local LLM runtime; the local option runs a repository-trained PyTorch seq2seq model and falls back to Ollama when no usable checkpoint exists.

• expanders/base.py

• expanders/ollama_expander.py

• expanders/local_expander.py

• config/api.yaml

Expansion is optional per request and is selected globally by API configuration, making it a pre-retrieval transformation rather than a retrieval-model concern.

Custom Expander Training and Model

The repository includes a from-scratch transformer encoder-decoder with attention and interchangeable tokenizers. Training data generation can use Ollama; training emits versioned checkpoints, from which the local expander selects the best validation-loss run.

• expander_train.py

• expander_model/expander_model.py

• expander_model/encoder.py

• expander_model/decoder.py

• config/expander_train.yaml

API and Client Boundary

FastAPI owns startup/lifespan initialization, access-code enforcement, REST endpoints, and static-file delivery. It coordinates recommender, expander, and database dependencies; browser UI logic is deliberately separate and served as static content.

• api.py

• static/index.html

• static/Privacy.html

• static/Terms.html

Material runtimes are Python/FastAPI, SQLite, PyTorch, Sentence Transformers, NumPy artifacts, and optionally Ollama. Configuration is YAML-based, with secrets supplied through environment variables.