How l3vith/paper-reccomendation works
This is an end-to-end research-paper discovery system: it gathers paper metadata, learns citation-aware embeddings, indexes the corpus, and serves local or live recommendations through a Streamlit interface. Scraping, triplet generation, training, evaluation, and indexing are distinct workflow stages; the recommendation layer combines local retrieval with optional live sources. The UI also exposes model benchmarks, corpus browsing, PDF reading, and speech playback. Edges reflect the README and shown excerpts or SOURCE INDEX references; unsampled internals are not assumed beyond those references.
Main entry points: recommend.py Recommendation service · app_streamlit.py Streamlit interface · pipeline.py Scrape orchestrator · train.py Embedding training
Components 21
Paper collection
- Scrape orchestrator
src/scraper/pipeline.py - arXiv source
src/scraper/arxiv_scraper.py - Semantic Scholar source
src/scraper/semantic_scholar_scraper.py - Firecrawl source
src/scraper/firecrawl_scraper.py - Paper corpus
src/scraper/data_store.py
Representation learning
- Citation triplets
src/model/build_triplets.py - Embedding training
src/model/train.py - Pooling variants
src/model/variants.py - Model evaluation
src/model/evaluate.py - Variant benchmarks
src/model/experiments.py
Search and recommendation
- FAISS indexer
src/recommender/indexer.py - Vector index
src/recommender/indexer.py - Query expansion
src/model/query_expansion.py - Recommendation service
src/recommender/recommend.py
User experience
- Streamlit interface
app_streamlit.py - PDF reader
src/pdf_reader.py - Paper downloader
src/downloader.py - Speech playback
src/tts_engine.py
Other components
- Researcher
- arXiv API
- Semantic Scholar API
Connections 27
- Researcher → Streamlit interface: uses
README.md - Scrape orchestrator → arXiv source: fetches papers
src/scraper/pipeline.py - Scrape orchestrator → Semantic Scholar source: fetches papers
src/scraper/pipeline.py - Scrape orchestrator → Firecrawl source: fetches papers
src/scraper/pipeline.py - arXiv source → arXiv API: queries
README.md - Semantic Scholar source → Semantic Scholar API: queries
src/scraper/semantic_scholar_scraper.py - Scrape orchestrator → Paper corpus: stores papers
src/scraper/pipeline.py - Citation triplets → Paper corpus: reads citations
src/model/build_triplets.py - Embedding training → Citation triplets: trains on
run_pipeline.py - Embedding training → Pooling variants: configures pooling
- Model evaluation → Embedding training: evaluates model
src/model/evaluate.py - Variant benchmarks → Pooling variants: compares
src/model/experiments.py - Variant benchmarks → Embedding training: fine-tunes
src/model/experiments.py - Streamlit interface → Variant benchmarks: runs benchmarks
app_streamlit.py - Streamlit interface → Recommendation service: requests recommendations
app_streamlit.py - Streamlit interface → PDF reader: opens PDFs
app_streamlit.py - Streamlit interface → Paper downloader: downloads papers
app_streamlit.py - Streamlit interface → Speech playback: plays speech
app_streamlit.py - FAISS indexer → Paper corpus: reads corpus
src/recommender/indexer.py - FAISS indexer → Vector index: builds index
README.md - Recommendation service → FAISS indexer: searches index
src/recommender/recommend.py - Recommendation service → Paper corpus: stores fetched papers
src/recommender/recommend.py - Recommendation service → Query expansion: expands queries
src/recommender/recommend.py - Recommendation service → arXiv source: refreshes candidates
src/recommender/recommend.py - Recommendation service → Semantic Scholar source: fetches sparse topics
src/recommender/recommend.py - Recommendation service → Firecrawl source: fetches fallback
src/recommender/recommend.py - Streamlit interface → Pooling variants: selects model
app_streamlit.py