# l3vith/paper-reccomendation architecture

> Architecture diagram of the GitHub repository l3vith/paper-reccomendation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/l3vith/paper-reccomendation
- Repository: https://github.com/l3vith/paper-reccomendation

## Overview

This is an end-to-end research-paper discovery system: it gathers paper metadata, learns citation-aware embeddings, indexes the corpus, and serves local or live recommendations through a Streamlit interface. Scraping, triplet generation, training, evaluation, and indexing are distinct workflow stages; the recommendation layer combines local retrieval with optional live sources. The UI also exposes model benchmarks, corpus browsing, PDF reading, and speech playback. Edges reflect the README and shown excerpts or SOURCE INDEX references; unsampled internals are not assumed beyond those references.

Main entry points: [`src/recommender/recommend.py`](https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/recommend.py) (Recommendation service), [`app_streamlit.py`](https://github.com/l3vith/paper-reccomendation/blob/main/app_streamlit.py) (Streamlit interface), [`src/scraper/pipeline.py`](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/pipeline.py) (Scrape orchestrator), [`src/model/train.py`](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/train.py) (Embedding training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Paper collection"]
  node_scrape_pipeline["Scrape orchestrator<br/>[pipeline.py]"]
  node_arxiv["arXiv source<br/>[arxiv_scraper.py]"]
  node_semantic["Semantic Scholar source"]
  node_firecrawl["Firecrawl source"]
  node_papers_db[("Paper corpus<br/>[data_store.py]")]
end

subgraph group_learning["Representation learning"]
  node_triplets["Citation triplets<br/>[build_triplets.py]"]
  node_training["Embedding training<br/>[train.py]"]
  node_variants["Pooling variants<br/>[variants.py]"]
  node_evaluation["Model evaluation<br/>[evaluate.py]"]
  node_benchmarks["Variant benchmarks<br/>[experiments.py]"]
end

subgraph group_discovery["Search and recommendation"]
  node_indexer["FAISS indexer<br/>[indexer.py]"]
  node_faiss[("Vector index<br/>[indexer.py]")]
  node_query_expansion["Query expansion<br/>[query_expansion.py]"]
  node_recommend["Recommendation service<br/>[recommend.py]"]
end

subgraph group_interface["User experience"]
  node_streamlit["Streamlit interface<br/>[app_streamlit.py]"]
  node_pdf_reader["PDF reader<br/>[pdf_reader.py]"]
  node_downloader["Paper downloader<br/>[downloader.py]"]
  node_tts["Speech playback<br/>[tts_engine.py]"]
end

node_user(("Researcher"))
node_arxiv_api["arXiv API"]
node_semantic_api["Semantic Scholar API"]

node_user -->|"uses"| node_streamlit
node_scrape_pipeline -->|"fetches papers"| node_arxiv
node_scrape_pipeline -->|"fetches papers"| node_semantic
node_scrape_pipeline -->|"fetches papers"| node_firecrawl
node_arxiv -->|"queries"| node_arxiv_api
node_semantic -->|"queries"| node_semantic_api
node_scrape_pipeline -->|"stores papers"| node_papers_db
node_triplets -->|"reads citations"| node_papers_db
node_training -->|"trains on"| node_triplets
node_training -->|"configures pooling"| node_variants
node_evaluation -->|"evaluates model"| node_training
node_benchmarks -->|"compares"| node_variants
node_benchmarks -->|"fine-tunes"| node_training
node_streamlit -->|"runs benchmarks"| node_benchmarks
node_streamlit -->|"requests recommendations"| node_recommend
node_streamlit -->|"opens PDFs"| node_pdf_reader
node_streamlit -->|"downloads papers"| node_downloader
node_streamlit -->|"plays speech"| node_tts
node_indexer -->|"reads corpus"| node_papers_db
node_indexer -->|"builds index"| node_faiss
node_recommend -->|"searches index"| node_indexer
node_recommend -->|"stores fetched papers"| node_papers_db
node_recommend -->|"expands queries"| node_query_expansion
node_recommend -->|"refreshes candidates"| node_arxiv
node_recommend -->|"fetches sparse topics"| node_semantic
node_recommend -->|"fetches fallback"| node_firecrawl
node_streamlit -->|"selects model"| node_variants

click node_streamlit "https://github.com/l3vith/paper-reccomendation/blob/main/app_streamlit.py"
click node_scrape_pipeline "https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/pipeline.py"
click node_arxiv "https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/arxiv_scraper.py"
click node_semantic "https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/semantic_scholar_scraper.py"
click node_firecrawl "https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/firecrawl_scraper.py"
click node_papers_db "https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/data_store.py"
click node_triplets "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/build_triplets.py"
click node_training "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/train.py"
click node_variants "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/variants.py"
click node_evaluation "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/evaluate.py"
click node_benchmarks "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/experiments.py"
click node_indexer "https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/indexer.py"
click node_faiss "https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/indexer.py"
click node_query_expansion "https://github.com/l3vith/paper-reccomendation/blob/main/src/model/query_expansion.py"
click node_recommend "https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/recommend.py"
click node_pdf_reader "https://github.com/l3vith/paper-reccomendation/blob/main/src/pdf_reader.py"
click node_downloader "https://github.com/l3vith/paper-reccomendation/blob/main/src/downloader.py"
click node_tts "https://github.com/l3vith/paper-reccomendation/blob/main/src/tts_engine.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scrape_pipeline,node_arxiv,node_semantic,node_firecrawl,node_papers_db toneBlue
class node_triplets,node_training,node_variants,node_evaluation,node_benchmarks toneAmber
class node_indexer,node_faiss,node_query_expansion,node_recommend,node_arxiv_api,node_semantic_api toneMint
class node_streamlit,node_pdf_reader,node_downloader,node_tts toneRose
class node_user toneIndigo
```

## Components

### Paper collection

- [Scrape orchestrator](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/pipeline.py): `src/scraper/pipeline.py`
- [arXiv source](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/arxiv_scraper.py): `src/scraper/arxiv_scraper.py`
- [Semantic Scholar source](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/semantic_scholar_scraper.py): `src/scraper/semantic_scholar_scraper.py`
- [Firecrawl source](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/firecrawl_scraper.py): `src/scraper/firecrawl_scraper.py`
- [Paper corpus](https://github.com/l3vith/paper-reccomendation/blob/main/src/scraper/data_store.py): `src/scraper/data_store.py`

### Representation learning

- [Citation triplets](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/build_triplets.py): `src/model/build_triplets.py`
- [Embedding training](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/train.py): `src/model/train.py`
- [Pooling variants](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/variants.py): `src/model/variants.py`
- [Model evaluation](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/evaluate.py): `src/model/evaluate.py`
- [Variant benchmarks](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/experiments.py): `src/model/experiments.py`

### Search and recommendation

- [FAISS indexer](https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/indexer.py): `src/recommender/indexer.py`
- [Vector index](https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/indexer.py): `src/recommender/indexer.py`
- [Query expansion](https://github.com/l3vith/paper-reccomendation/blob/main/src/model/query_expansion.py): `src/model/query_expansion.py`
- [Recommendation service](https://github.com/l3vith/paper-reccomendation/blob/main/src/recommender/recommend.py): `src/recommender/recommend.py`

### User experience

- [Streamlit interface](https://github.com/l3vith/paper-reccomendation/blob/main/app_streamlit.py): `app_streamlit.py`
- [PDF reader](https://github.com/l3vith/paper-reccomendation/blob/main/src/pdf_reader.py): `src/pdf_reader.py`
- [Paper downloader](https://github.com/l3vith/paper-reccomendation/blob/main/src/downloader.py): `src/downloader.py`
- [Speech playback](https://github.com/l3vith/paper-reccomendation/blob/main/src/tts_engine.py): `src/tts_engine.py`

### Other components

- **Researcher**
- **arXiv API**
- **Semantic Scholar API**

## Connections

- Researcher → Streamlit interface: uses (evidence: `README.md`)
- Scrape orchestrator → arXiv source: fetches papers (evidence: `src/scraper/pipeline.py`)
- Scrape orchestrator → Semantic Scholar source: fetches papers (evidence: `src/scraper/pipeline.py`)
- Scrape orchestrator → Firecrawl source: fetches papers (evidence: `src/scraper/pipeline.py`)
- arXiv source → arXiv API: queries (evidence: `README.md`)
- Semantic Scholar source → Semantic Scholar API: queries (evidence: `src/scraper/semantic_scholar_scraper.py`)
- Scrape orchestrator → Paper corpus: stores papers (evidence: `src/scraper/pipeline.py`)
- Citation triplets → Paper corpus: reads citations (evidence: `src/model/build_triplets.py`)
- Embedding training → Citation triplets: trains on (evidence: `run_pipeline.py`)
- Embedding training → Pooling variants: configures pooling
- Model evaluation → Embedding training: evaluates model (evidence: `src/model/evaluate.py`)
- Variant benchmarks → Pooling variants: compares (evidence: `src/model/experiments.py`)
- Variant benchmarks → Embedding training: fine-tunes (evidence: `src/model/experiments.py`)
- Streamlit interface → Variant benchmarks: runs benchmarks (evidence: `app_streamlit.py`)
- Streamlit interface → Recommendation service: requests recommendations (evidence: `app_streamlit.py`)
- Streamlit interface → PDF reader: opens PDFs (evidence: `app_streamlit.py`)
- Streamlit interface → Paper downloader: downloads papers (evidence: `app_streamlit.py`)
- Streamlit interface → Speech playback: plays speech (evidence: `app_streamlit.py`)
- FAISS indexer → Paper corpus: reads corpus (evidence: `src/recommender/indexer.py`)
- FAISS indexer → Vector index: builds index (evidence: `README.md`)
- Recommendation service → FAISS indexer: searches index (evidence: `src/recommender/recommend.py`)
- Recommendation service → Paper corpus: stores fetched papers (evidence: `src/recommender/recommend.py`)
- Recommendation service → Query expansion: expands queries (evidence: `src/recommender/recommend.py`)
- Recommendation service → arXiv source: refreshes candidates (evidence: `src/recommender/recommend.py`)
- Recommendation service → Semantic Scholar source: fetches sparse topics (evidence: `src/recommender/recommend.py`)
- Recommendation service → Firecrawl source: fetches fallback (evidence: `src/recommender/recommend.py`)
- Streamlit interface → Pooling variants: selects model (evidence: `app_streamlit.py`)
