# yash2010/recommendation_system architecture

> Architecture diagram of the GitHub repository yash2010/recommendation_system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yash2010/recommendation_system
- Repository: https://github.com/yash2010/recommendation_system

## Overview

#### Purpose and Entry Points
This is a semantic movie recommendation service: it ingests TMDB metadata, stores movies and user interactions in SQLite, builds a sentence-embedding index, and serves search/recommendation APIs plus a static UI. Runtime entry point is `api.py`; offline operational entry points are `scripts/fetch_tmdb.py`, `scripts/csv2db.py`, `scripts/build_index.py`, and `expander_train.py`.

#### Data Acquisition and Preparation
The TMDB pipeline fetches multilingual movie records, applies filtering/normalization, and exports checkpointed datasets for database import. TMDB is an external dependency authenticated by `TMDB_TOKEN`.
- `data_pipeline/pipeline.py`
- `data_pipeline/fetcher.py`
- `data_pipeline/processor.py`
- `data_pipeline/exporter.py`
- `data_pipeline/config.yaml`

The intended boundary is file-based: pipeline output (CSV or Parquet) is imported into SQLite separately rather than queried directly by the serving layer.

#### Persistent Catalog and Feedback Store
SQLite is the system of record for movie metadata, users, and interactions. The database layer supports feedback/history needed for eventual personalization, while the current retrieval path remains content-based.
- `database.py`
- `scripts/csv2db.py`

User interaction writes (`click`, `rate`, `watch`) enter through the API and are persisted independently of embedding-index generation.

#### Embedding Index and Retrieval
Index construction embeds each movie using the external Sentence Transformers model `all-MiniLM-L6-v2`, producing artifact files aligned with movie metadata. At serving time, the recommender embeds the query and performs cosine-similarity ranking over the loaded NumPy embedding matrix; it also supports title lookup, genre filtering, and item-to-item similarity.
- `scripts/build_index.py`
- `recommender.py`
- `api.py`

This creates a deployment consistency boundary: SQLite catalog contents and `artifacts/embeddings.npy` must be rebuilt/kept aligned after catalog changes.

#### Query Expansion Subsystem
Before semantic retrieval, requests may be enriched through a pluggable expander abstraction. Ollama is an external local LLM runtime; the local option runs a repository-trained PyTorch seq2seq model and falls back to Ollama when no usable checkpoint exists.
- `expanders/base.py`
- `expanders/ollama_expander.py`
- `expanders/local_expander.py`
- `config/api.yaml`

Expansion is optional per request and is selected globally by API configuration, making it a pre-retrieval transformation rather than a retrieval-model concern.

#### Custom Expander Training and Model
The repository includes a from-scratch transformer encoder-decoder with attention and interchangeable tokenizers. Training data generation can use Ollama; training emits versioned checkpoints, from which the local expander selects the best validation-loss run.
- `expander_train.py`
- `expander_model/expander_model.py`
- `expander_model/encoder.py`
- `expander_model/decoder.py`
- `config/expander_train.yaml`

#### API and Client Boundary
FastAPI owns startup/lifespan initialization, access-code enforcement, REST endpoints, and static-file delivery. It coordinates recommender, expander, and database dependencies; browser UI logic is deliberately separate and served as static content.
- `api.py`
- `static/index.html`
- `static/Privacy.html`
- `static/Terms.html`

Material runtimes are Python/FastAPI, SQLite, PyTorch, Sentence Transformers, NumPy artifacts, and optionally Ollama. Configuration is YAML-based, with secrets supplied through environment variables.

Main entry points: [`recommender.py`](https://github.com/yash2010/recommendation_system/blob/main/recommender.py) (Semantic recommender), [`api.py`](https://github.com/yash2010/recommendation_system/blob/main/api.py) (FastAPI service), [`database.py`](https://github.com/yash2010/recommendation_system/blob/main/database.py) (Movie and feedback store), [`expanders/local_expander.py`](https://github.com/yash2010/recommendation_system/blob/main/expanders/local_expander.py) (Pluggable query expander)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Catalog ingestion"]
  node_tmdb(("TMDB API<br/>external movie source"))
  node_pipeline["TMDB pipeline<br/>Python pipeline<br/>[pipeline.py]"]
  node_fetcher["Multilingual fetcher<br/>pipeline stage<br/>[fetcher.py]"]
  node_processor["Filter and normalize<br/>pipeline stage<br/>[processor.py]"]
  node_exporter["Dataset exporter<br/>pipeline stage<br/>[exporter.py]"]
  node_catalog_files["Catalog datasets<br/>CSV / Parquet"]
  node_csv_import["CSV-to-SQLite import<br/>offline script<br/>[csv2db.py]"]
end

subgraph group_indexing["Embedding index"]
  node_index_builder["Index builder<br/>offline script<br/>[build_index.py]"]
  node_sentence_model{{"all-MiniLM-L6-v2<br/>Sentence Transformers"}}
  node_embeddings["Embedding matrix<br/>NumPy artifact"]
end

subgraph group_expansion["Query expansion"]
  node_ollama{{"Ollama<br/>local LLM runtime"}}
  node_expander_training["Expander training<br/>PyTorch training<br/>[expander_train.py]"]
  node_seq2seq_model["Custom seq2seq transformer<br/>PyTorch model<br/>[expander_model.py]"]
  node_checkpoints["Expander checkpoints<br/>model artifacts"]
  node_expander["Pluggable query expander<br/>pre-retrieval service<br/>[local_expander.py]"]
end

subgraph group_serving["API and client"]
  node_recommender["Semantic recommender<br/>retrieval service<br/>[recommender.py]"]
  node_api["FastAPI service<br/>Python API<br/>[api.py]"]
  node_static_ui["Static browser UI<br/>HTML client<br/>[index.html]"]
end

node_sqlite[("Movie and feedback store<br/>SQLite<br/>[database.py]")]

node_tmdb -->|"movie records"| node_fetcher
node_fetcher -->|"raw metadata"| node_processor
node_processor -->|"normalized movies"| node_exporter
node_pipeline -->|"orchestrates"| node_fetcher
node_exporter -->|"exports"| node_catalog_files
node_catalog_files -->|"imports"| node_csv_import
node_csv_import -->|"catalog rows"| node_sqlite
node_sqlite -->|"movie metadata"| node_index_builder
node_sentence_model -->|"embeds movies"| node_index_builder
node_index_builder -->|"builds"| node_embeddings
node_ollama -.->|"optional training data"| node_expander_training
node_seq2seq_model -->|"trains"| node_expander_training
node_expander_training -->|"writes"| node_checkpoints
node_checkpoints -->|"best checkpoint"| node_expander
node_ollama -.->|"fallback expansion"| node_expander
node_expander -->|"expanded query"| node_recommender
node_sentence_model -->|"embeds query"| node_recommender
node_embeddings -->|"similarity matrix"| node_recommender
node_sqlite -->|"catalog metadata"| node_recommender
node_api -->|"optional request expansion"| node_expander
node_api -->|"search and recommendations"| node_recommender
node_api -->|"click, rate, watch writes"| node_sqlite
node_api -->|"serves"| node_static_ui

click node_pipeline "https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/pipeline.py"
click node_fetcher "https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/fetcher.py"
click node_processor "https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/processor.py"
click node_exporter "https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/exporter.py"
click node_csv_import "https://github.com/yash2010/recommendation_system/blob/main/scripts/csv2db.py"
click node_sqlite "https://github.com/yash2010/recommendation_system/blob/main/database.py"
click node_index_builder "https://github.com/yash2010/recommendation_system/blob/main/scripts/build_index.py"
click node_expander_training "https://github.com/yash2010/recommendation_system/blob/main/expander_train.py"
click node_seq2seq_model "https://github.com/yash2010/recommendation_system/blob/main/expander_model/expander_model.py"
click node_expander "https://github.com/yash2010/recommendation_system/blob/main/expanders/local_expander.py"
click node_recommender "https://github.com/yash2010/recommendation_system/blob/main/recommender.py"
click node_api "https://github.com/yash2010/recommendation_system/blob/main/api.py"
click node_static_ui "https://github.com/yash2010/recommendation_system/blob/main/static/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tmdb,node_pipeline,node_fetcher,node_processor,node_exporter,node_catalog_files,node_csv_import toneBlue
class node_index_builder,node_sentence_model,node_embeddings toneAmber
class node_ollama,node_expander_training,node_seq2seq_model,node_checkpoints,node_expander toneMint
class node_recommender,node_api,node_static_ui toneRose
class node_sqlite toneNeutral
```

## Components

### Catalog ingestion

- **TMDB API**: external movie source · Authenticated with TMDB_TOKEN.
- [TMDB pipeline](https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/pipeline.py): Python pipeline · `data_pipeline/pipeline.py`
- [Multilingual fetcher](https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/fetcher.py): pipeline stage · `data_pipeline/fetcher.py`
- [Filter and normalize](https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/processor.py): pipeline stage · `data_pipeline/processor.py`
- [Dataset exporter](https://github.com/yash2010/recommendation_system/blob/main/data_pipeline/exporter.py): pipeline stage · Writes checkpointed CSV or Parquet for separate import. · `data_pipeline/exporter.py`
- **Catalog datasets**: CSV / Parquet
- [CSV-to-SQLite import](https://github.com/yash2010/recommendation_system/blob/main/scripts/csv2db.py): offline script · `scripts/csv2db.py`

### Embedding index

- [Index builder](https://github.com/yash2010/recommendation_system/blob/main/scripts/build_index.py): offline script · Must be rerun when catalog contents change. · `scripts/build_index.py`
- **all-MiniLM-L6-v2**: Sentence Transformers
- **Embedding matrix**: NumPy artifact · Aligned with the SQLite movie catalog.

### Query expansion

- **Ollama**: local LLM runtime · Used for expansion and optional training-data generation.
- [Expander training](https://github.com/yash2010/recommendation_system/blob/main/expander_train.py): PyTorch training · Emits versioned checkpoints selected by validation loss. · `expander_train.py`
- [Custom seq2seq transformer](https://github.com/yash2010/recommendation_system/blob/main/expander_model/expander_model.py): PyTorch model · `expander_model/expander_model.py`
- **Expander checkpoints**: model artifacts
- [Pluggable query expander](https://github.com/yash2010/recommendation_system/blob/main/expanders/local_expander.py): pre-retrieval service · Uses local checkpoints or falls back to Ollama. · `expanders/local_expander.py`

### API and client

- [Semantic recommender](https://github.com/yash2010/recommendation_system/blob/main/recommender.py): retrieval service · Ranks query embeddings and supports title, genre, and similar-item lookup. · `recommender.py`
- [FastAPI service](https://github.com/yash2010/recommendation_system/blob/main/api.py): Python API · Owns startup, access-code enforcement, REST, and static delivery. · `api.py`
- [Static browser UI](https://github.com/yash2010/recommendation_system/blob/main/static/index.html): HTML client · `static/index.html`

### Other components

- [Movie and feedback store](https://github.com/yash2010/recommendation_system/blob/main/database.py): SQLite · System of record for catalog, users, and interactions. · `database.py`

## Connections

- TMDB API → Multilingual fetcher: movie records
- Multilingual fetcher → Filter and normalize: raw metadata
- Filter and normalize → Dataset exporter: normalized movies
- TMDB pipeline → Multilingual fetcher: orchestrates
- Dataset exporter → Catalog datasets: exports
- Catalog datasets → CSV-to-SQLite import: imports
- CSV-to-SQLite import → Movie and feedback store: catalog rows
- Movie and feedback store → Index builder: movie metadata
- all-MiniLM-L6-v2 → Index builder: embeds movies
- Index builder → Embedding matrix: builds
- Ollama → Expander training: optional training data
- Custom seq2seq transformer → Expander training: trains
- Expander training → Expander checkpoints: writes
- Expander checkpoints → Pluggable query expander: best checkpoint
- Ollama → Pluggable query expander: fallback expansion
- Pluggable query expander → Semantic recommender: expanded query
- all-MiniLM-L6-v2 → Semantic recommender: embeds query
- Embedding matrix → Semantic recommender: similarity matrix
- Movie and feedback store → Semantic recommender: catalog metadata
- FastAPI service → Pluggable query expander: optional request expansion
- FastAPI service → Semantic recommender: search and recommendations
- FastAPI service → Movie and feedback store: click, rate, watch writes
- FastAPI service → Static browser UI: serves