How srivenkateshmani/paperpilot works
PaperPilot’s principal documented workflow ingests arXiv papers into searchable chunks, then retrieves evidence for grounded, cited answers. The tree supports ingestion, BM25/vector search, hybrid retrieval, and a separate metadata API. The README describes Airflow scheduling and LangGraph-based iterative RAG, but those implementations are not present in the supplied tree; their links are documented rather than source-verified.
Main entry points: run_ingestion.py Ingestion Runner · hybrid_search.py Hybrid Retrieval · papers.py Paper Routes · store_papers.py Metadata Storage
Components 22
Paper Ingestion
- Ingestion Runner
app/ingestion/run_ingestion.py - Paper Search
app/ingestion/arxiv_client.py - Metadata Storage
app/ingestion/store_papers.py - PDF Download
app/ingestion/pdf_downloader.py - PDF Parsing
app/ingestion/pdf_parser.py - Text Chunking
app/ingestion/chunker.py - Embedding Model
app/ingestion/embedder.py - Chunk Indexing
app/ingestion/indexer.py - PostgreSQL
app/database.py - Logging
app/logger.py
Search and Retrieval
- OpenSearch Index
app/ingestion/indexer.py - BM25 Search
app/retrieve/bm25_search.py - Vector Search
app/retrieve/semantic_search.py - Hybrid Retrieval
app/retrieve/hybrid_search.py
Grounded Answering
- Agentic Retrieval
- Grounded Answer
Metadata API
- FastAPI Application
app/main.py - Paper Routes
app/routes/papers.py - Paper Model
app/models.py - Paper Schema
app/schemas.py
Other components
- Researcher
- arXiv
Connections 24
- Ingestion Runner → Metadata Storage: calls
app/ingestion/run_ingestion.py - Metadata Storage → Paper Search: searches
app/ingestion/store_papers.py - Paper Search → arXiv: queries
app/ingestion/arxiv_client.py - Metadata Storage → PostgreSQL: stores metadata
app/ingestion/store_papers.py - Ingestion Runner → PDF Download: calls
app/ingestion/run_ingestion.py - PDF Download → PDF Parsing
- Ingestion Runner → PDF Parsing: calls
app/ingestion/run_ingestion.py - Ingestion Runner → Text Chunking: calls
app/ingestion/run_ingestion.py - Ingestion Runner → Embedding Model: calls
app/ingestion/run_ingestion.py - Ingestion Runner → Chunk Indexing: indexes chunks
app/ingestion/run_ingestion.py - Chunk Indexing → OpenSearch Index: writes chunks
app/ingestion/indexer.py - Researcher → Hybrid Retrieval: asks query
README.md - Hybrid Retrieval → BM25 Search: runs keyword search
app/retrieve/hybrid_search.py - Hybrid Retrieval → Vector Search: runs vector search
app/retrieve/hybrid_search.py - BM25 Search → OpenSearch Index: searches
app/retrieve/bm25_search.py - Vector Search → OpenSearch Index: searches
app/retrieve/semantic_search.py - Vector Search → Embedding Model: encodes query
app/retrieve/semantic_search.py - Hybrid Retrieval → Agentic Retrieval: feeds evidence
README.md - Agentic Retrieval → Grounded Answer: generates cited answer
README.md - FastAPI Application → Paper Routes: registers routes
app/main.py - FastAPI Application → PostgreSQL: creates tables
app/main.py - Paper Routes → PostgreSQL: reads and writes
app/routes/papers.py - Paper Routes → Paper Model: uses model
app/routes/papers.py - Paper Routes → Paper Schema: validates input
app/routes/papers.py