# t-redactyl/llm-semantic-book-recommender architecture

> Architecture diagram of the GitHub repository t-redactyl/llm-semantic-book-recommender, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/t-redactyl/llm-semantic-book-recommender
- Repository: https://github.com/t-redactyl/llm-semantic-book-recommender

## Overview

This is a **Python ML/LLM prototype + tutorial repo** that builds a **semantic book recommender**. Architecturally, it’s best represented as two “modes”:

##### Offline / batch pipeline (notebooks) that
- downloads + cleans data
- builds embeddings and a **vector database (Chroma via LangChain)**
- derives metadata facets (fiction/non-fiction via zero-shot classification)
- derives sentiment/emotion scores (LLM-driven sentiment analysis)

##### Online / interactive app (Gradio) that
- accepts a natural-language query + filters/sort options
- performs vector search against the persisted vector DB
- returns ranked recommendations (optionally using the precomputed facets)

##### What the file tree implies (map files to architectural components)
Top-level files are “the architecture” here—there’s no src/ package layering. Each notebook is effectively a pipeline stage.

- `data-exploration.ipynb`
**ETL / data cleaning** + exploration; likely produces cleaned dataset artifacts (CSV/parquet) used downstream.

- `vector-search.ipynb`
**Embedding + indexing** stage; creates/persists a **Chroma vector database** via `langchain-chroma`. This is the core retrieval component used by the app.

- `text-classification.ipynb`
**Zero-shot classification** stage (Transformers or LLM). Output is likely per-book labels (fiction/non-fiction) used as **metadata filters**.

- `sentiment-analysis.ipynb`
**Emotion/sentiment extraction** stage; output is per-book emotion scores (suspenseful/joyful/sad, etc.) used for sorting/facets.

- `gradio-dashboard.py`
The **serving layer** (a small “backend” + UI). It loads the vector DB and metadata and exposes search/recommendation interactions via **Gradio**.

- `requirements.txt`
Captures the runtime ecosystem: Kaggle download (`kagglehub`), data libs, LangChain integrations, Chroma, Transformers, Gradio.

- `.env` (implied by README)

##### Key architecture/tech to highlight (from README)
- **Kaggle** as the data source (downloaded via `kagglehub`)
- **OpenAI API** (embeddings and/or LLM calls; configured via `.env`)
- **LangChain** (orchestrates embedding + retrieval + maybe LLM calls)
- **Chroma** as the **vector database** (local persistent store is the likely deployment)
- **Transformers** for zero-shot classification (depending on notebook implementation)
- **Gradio** as the web UI/app container

Architectural pattern:
- A classic **RAG-style retrieval** workflow (query → embedding → vector search → results), augmented with **metadata enrichment** (classification + sentiment) to support **filtering/sorting**.

##### Ingestion & Cleaning (`data-exploration.ipynb`, pandas)
3. **Embedding + Vector Index Builder** (`vector-search.ipynb`, LangChain + OpenAI embeddings)
##### Artifacts Store
- Cleaned dataset files (CSV/parquet)
- Enrichment outputs (labels, emotion scores)
- (Optionally combined metadata table)

**Storage / Retrieval**
##### Vector DB: Chroma (persisted locally)
Stores embeddings + document text + metadata.

**Online / App**
##### Gradio Web App (`gradio-dashboard.py`)
UI + orchestration to query Chroma and present recommendations.

**External Services**

##### Important design principles (annotate)
- **Batch preprocessing + offline indexing** (notebooks) vs **online retrieval** (Gradio)
- **Vector search** is the core retrieval mechanism (semantic similarity)
- **Metadata facets** enable filtering (fiction/non-fiction) and ranking by tone
- **Secrets management** via `.env` (OpenAI key)
- **Persistence boundary**: Chroma local persistence (stateful storage)

##### Relevant technologies (label inside components)
- “pandas / seaborn / matplotlib” in cleaning/exploration
- “LangChain + Chroma” in vector store stage
- “OpenAI Embeddings” (if used) on the embedding edge
- “Transformers zero-shot” on classification (or “OpenAI LLM” if used)

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% External (Cloud)
  %% =========================
  subgraph "External Services"
    direction TB
    Kaggle["Kaggle Dataset (Books)"]:::external
    OpenAI["OpenAI API (Embeddings / LLM)"]:::external
    HF["Transformers Models (Zero-shot / NLP)"]:::external
  end

  %% =========================
  %% Offline / Build-time
  %% =========================
  subgraph "Offline / Build Time (Notebooks Pipeline)"
    direction TB

    N1["Notebook: Data Ingestion & Cleaning\n(pandas / exploration)"]:::compute
    A1[("Cleaned Books Dataset\n(CSV / parquet artifacts)")]:::storage

    N2["Notebook: Embedding + Vector Index Builder\n(LangChain + OpenAI embeddings)"]:::compute
    N3["Notebook: Zero-shot Text Classification\n(fiction / non-fiction facets)"]:::compute
    N4["Notebook: Sentiment / Emotion Extraction\n(emotion scores for ranking)"]:::compute

    A2[("Metadata Artifacts\n(labels + emotion scores)")]:::storage
  end

  %% =========================
  %% Storage / Retrieval
  %% =========================
  subgraph "Storage / Retrieval Layer"
    direction TB
    Chroma[("Chroma Vector DB\n(persisted on disk)\nVectors + Docs + Metadata")]:::storage
  end

  %% =========================
  %% Online / Run-time
  %% =========================
  subgraph "Online / Run Time (Interactive App)"
    direction TB
    User["User Browser"]:::user
    Gradio["Gradio Dashboard\n(UI + orchestration)"]:::compute

    Note1["Note: Offline indexing builds state.\nOnline app performs retrieval per query."]:::note
  end

  %% =========================
  %% Provide nodes for mapped docs/deps without cluttering core flow
  %% =========================
  subgraph "Repo Files (Reference)"
    direction TB
    Readme["README (Usage + Architecture Notes)"]:::doc
    Req["requirements.txt (Runtime Dependencies)"]:::doc
  end

  %% =========================
  %% Offline data flow
  %% =========================
  Kaggle -->|"download_dataset (kagglehub)"| N1
  N1 -->|"clean_normalize_export"| A1

  A1 -->|"load_cleaned_data"| N2
  N2 -->|"embed_documents"| OpenAI
  N2 -->|"upsert_vectors+docs"| Chroma

  A1 -->|"load_text"| N3
  N3 -->|"zero-shot_infer"| HF
  N3 -.->|"optional_LLM_classify"| OpenAI
  N3 -->|"write_labels"| A2

  A1 -->|"load_text"| N4
  N4 -->|"emotion_extract"| HF
  N4 -.->|"optional_LLM_sentiment"| OpenAI
  N4 -->|"write_scores"| A2

  A2 -->|"attach_metadata_fields"| Chroma

  %% =========================
  %% Online query flow
  %% =========================
  User -->|"query+filters+sort"| Gradio
  Gradio -.->|"embed_query (if_live)"| OpenAI
  Gradio -->|"semantic_search_top-k\n+metadata_filters"| Chroma
  Chroma -->|"results:books+metadata"| Gradio
  Gradio -->|"ranked_recommendations"| User

  Note1 -.->|"highlights_boundary"| Gradio
  Note1 -.->|"highlights_persistence"| Chroma

  Readme -.->|"guides_setup"| Gradio
  Req -.->|"pins_libs"| N1
  Req -.->|"pins_libs"| N2
  Req -.->|"pins_libs"| N3
  Req -.->|"pins_libs"| N4

  %% =========================
  %% Click events (component mapping)
  %% =========================
  click N1 "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/data-exploration.ipynb"
  click N2 "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/vector-search.ipynb"
  click N3 "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/text-classification.ipynb"
  click N4 "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/sentiment-analysis.ipynb"
  click Gradio "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/gradio-dashboard.py"
  click Req "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/requirements.txt"
  click Readme "https://github.com/t-redactyl/llm-semantic-book-recommender/blob/main/README.md"

  %% =========================
  %% Styles
  %% =========================
  classDef external fill:#E5E7EB,stroke:#374151,stroke-width:1px,color:#111827
  classDef compute fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1B3A
  classDef storage fill:#DCFCE7,stroke:#16A34A,stroke-width:1px,color:#052E16
  classDef user fill:#FFE4E6,stroke:#BE123C,stroke-width:1px,color:#4C0519
  classDef note fill:#FEF9C3,stroke:#CA8A04,stroke-width:1px,color:#422006,stroke-dasharray: 4 3
  classDef doc fill:#F3E8FF,stroke:#7C3AED,stroke-width:1px,color:#2E1065
```
