# srivenkateshmani/paperpilot architecture

> Architecture diagram of the GitHub repository srivenkateshmani/paperpilot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/srivenkateshmani/paperpilot
- Repository: https://github.com/srivenkateshmani/paperpilot

## Overview

PaperPilot’s principal documented workflow ingests arXiv papers into searchable chunks, then retrieves evidence for grounded, cited answers. The tree supports ingestion, BM25/vector search, hybrid retrieval, and a separate metadata API. The README describes Airflow scheduling and LangGraph-based iterative RAG, but those implementations are not present in the supplied tree; their links are documented rather than source-verified.

Main entry points: [`app/ingestion/run_ingestion.py`](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/run_ingestion.py) (Ingestion Runner), [`app/retrieve/hybrid_search.py`](https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/hybrid_search.py) (Hybrid Retrieval), [`app/routes/papers.py`](https://github.com/srivenkateshmani/paperpilot/blob/main/app/routes/papers.py) (Paper Routes), [`app/ingestion/store_papers.py`](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/store_papers.py) (Metadata Storage)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Paper Ingestion"]
  node_ingest_runner["Ingestion Runner<br/>[run_ingestion.py]"]
  node_paper_search["Paper Search<br/>[arxiv_client.py]"]
  node_paper_metadata["Metadata Storage<br/>[store_papers.py]"]
  node_pdf_download["PDF Download<br/>[pdf_downloader.py]"]
  node_pdf_parse["PDF Parsing<br/>[pdf_parser.py]"]
  node_chunking["Text Chunking<br/>[chunker.py]"]
  node_embedding["Embedding Model<br/>[embedder.py]"]
  node_indexing["Chunk Indexing<br/>[indexer.py]"]
  node_postgres[("PostgreSQL<br/>[database.py]")]
  node_logger["Logging<br/>[logger.py]"]
end

subgraph group_retrieval["Search and Retrieval"]
  node_opensearch[("OpenSearch Index<br/>[indexer.py]")]
  node_bm25["BM25 Search<br/>[bm25_search.py]"]
  node_semantic["Vector Search<br/>[semantic_search.py]"]
  node_hybrid["Hybrid Retrieval<br/>[hybrid_search.py]"]
end

subgraph group_answering["Grounded Answering"]
  node_langgraph["Agentic Retrieval"]
  node_answer["Grounded Answer"]
end

subgraph group_api["Metadata API"]
  node_api_main["FastAPI Application<br/>[main.py]"]
  node_paper_routes["Paper Routes<br/>[papers.py]"]
  node_paper_model["Paper Model<br/>[models.py]"]
  node_paper_schema["Paper Schema<br/>[schemas.py]"]
end

node_researcher(("Researcher"))
node_arxiv["arXiv"]

node_ingest_runner -->|"calls"| node_paper_metadata
node_paper_metadata -->|"searches"| node_paper_search
node_paper_search -->|"queries"| node_arxiv
node_paper_metadata -->|"stores metadata"| node_postgres
node_ingest_runner -->|"calls"| node_pdf_download
node_pdf_download -.-> node_pdf_parse
node_ingest_runner -->|"calls"| node_pdf_parse
node_ingest_runner -->|"calls"| node_chunking
node_ingest_runner -->|"calls"| node_embedding
node_ingest_runner -->|"indexes chunks"| node_indexing
node_indexing -->|"writes chunks"| node_opensearch
node_researcher -.->|"asks query"| node_hybrid
node_hybrid -->|"runs keyword search"| node_bm25
node_hybrid -->|"runs vector search"| node_semantic
node_bm25 -->|"searches"| node_opensearch
node_semantic -->|"searches"| node_opensearch
node_semantic -->|"encodes query"| node_embedding
node_hybrid -.->|"feeds evidence"| node_langgraph
node_langgraph -.->|"generates cited answer"| node_answer
node_api_main -->|"registers routes"| node_paper_routes
node_api_main -->|"creates tables"| node_postgres
node_paper_routes -->|"reads and writes"| node_postgres
node_paper_routes -->|"uses model"| node_paper_model
node_paper_routes -->|"validates input"| node_paper_schema

click node_ingest_runner "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/run_ingestion.py"
click node_paper_search "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/arxiv_client.py"
click node_paper_metadata "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/store_papers.py"
click node_pdf_download "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/pdf_downloader.py"
click node_pdf_parse "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/pdf_parser.py"
click node_chunking "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/chunker.py"
click node_embedding "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/embedder.py"
click node_indexing "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/indexer.py"
click node_postgres "https://github.com/srivenkateshmani/paperpilot/blob/main/app/database.py"
click node_opensearch "https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/indexer.py"
click node_bm25 "https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/bm25_search.py"
click node_semantic "https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/semantic_search.py"
click node_hybrid "https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/hybrid_search.py"
click node_api_main "https://github.com/srivenkateshmani/paperpilot/blob/main/app/main.py"
click node_paper_routes "https://github.com/srivenkateshmani/paperpilot/blob/main/app/routes/papers.py"
click node_paper_model "https://github.com/srivenkateshmani/paperpilot/blob/main/app/models.py"
click node_paper_schema "https://github.com/srivenkateshmani/paperpilot/blob/main/app/schemas.py"
click node_logger "https://github.com/srivenkateshmani/paperpilot/blob/main/app/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_runner,node_paper_search,node_paper_metadata,node_pdf_download,node_pdf_parse,node_chunking,node_embedding,node_indexing,node_postgres,node_logger toneBlue
class node_opensearch,node_bm25,node_semantic,node_hybrid toneAmber
class node_langgraph,node_answer toneMint
class node_api_main,node_paper_routes,node_paper_model,node_paper_schema toneRose
class node_researcher,node_arxiv toneIndigo
```

## Components

### Paper Ingestion

- [Ingestion Runner](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/run_ingestion.py): `app/ingestion/run_ingestion.py`
- [Paper Search](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/arxiv_client.py): `app/ingestion/arxiv_client.py`
- [Metadata Storage](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/store_papers.py): `app/ingestion/store_papers.py`
- [PDF Download](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/pdf_downloader.py): `app/ingestion/pdf_downloader.py`
- [PDF Parsing](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/pdf_parser.py): `app/ingestion/pdf_parser.py`
- [Text Chunking](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/chunker.py): `app/ingestion/chunker.py`
- [Embedding Model](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/embedder.py): `app/ingestion/embedder.py`
- [Chunk Indexing](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/indexer.py): `app/ingestion/indexer.py`
- [PostgreSQL](https://github.com/srivenkateshmani/paperpilot/blob/main/app/database.py): `app/database.py`
- [Logging](https://github.com/srivenkateshmani/paperpilot/blob/main/app/logger.py): `app/logger.py`

### Search and Retrieval

- [OpenSearch Index](https://github.com/srivenkateshmani/paperpilot/blob/main/app/ingestion/indexer.py): `app/ingestion/indexer.py`
- [BM25 Search](https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/bm25_search.py): `app/retrieve/bm25_search.py`
- [Vector Search](https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/semantic_search.py): `app/retrieve/semantic_search.py`
- [Hybrid Retrieval](https://github.com/srivenkateshmani/paperpilot/blob/main/app/retrieve/hybrid_search.py): `app/retrieve/hybrid_search.py`

### Grounded Answering

- **Agentic Retrieval**
- **Grounded Answer**

### Metadata API

- [FastAPI Application](https://github.com/srivenkateshmani/paperpilot/blob/main/app/main.py): `app/main.py`
- [Paper Routes](https://github.com/srivenkateshmani/paperpilot/blob/main/app/routes/papers.py): `app/routes/papers.py`
- [Paper Model](https://github.com/srivenkateshmani/paperpilot/blob/main/app/models.py): `app/models.py`
- [Paper Schema](https://github.com/srivenkateshmani/paperpilot/blob/main/app/schemas.py): `app/schemas.py`

### Other components

- **Researcher**
- **arXiv**

## Connections

- Ingestion Runner → Metadata Storage: calls (evidence: `app/ingestion/run_ingestion.py`)
- Metadata Storage → Paper Search: searches (evidence: `app/ingestion/store_papers.py`)
- Paper Search → arXiv: queries (evidence: `app/ingestion/arxiv_client.py`)
- Metadata Storage → PostgreSQL: stores metadata (evidence: `app/ingestion/store_papers.py`)
- Ingestion Runner → PDF Download: calls (evidence: `app/ingestion/run_ingestion.py`)
- PDF Download → PDF Parsing
- Ingestion Runner → PDF Parsing: calls (evidence: `app/ingestion/run_ingestion.py`)
- Ingestion Runner → Text Chunking: calls (evidence: `app/ingestion/run_ingestion.py`)
- Ingestion Runner → Embedding Model: calls (evidence: `app/ingestion/run_ingestion.py`)
- Ingestion Runner → Chunk Indexing: indexes chunks (evidence: `app/ingestion/run_ingestion.py`)
- Chunk Indexing → OpenSearch Index: writes chunks (evidence: `app/ingestion/indexer.py`)
- Researcher → Hybrid Retrieval: asks query (evidence: `README.md`)
- Hybrid Retrieval → BM25 Search: runs keyword search (evidence: `app/retrieve/hybrid_search.py`)
- Hybrid Retrieval → Vector Search: runs vector search (evidence: `app/retrieve/hybrid_search.py`)
- BM25 Search → OpenSearch Index: searches (evidence: `app/retrieve/bm25_search.py`)
- Vector Search → OpenSearch Index: searches (evidence: `app/retrieve/semantic_search.py`)
- Vector Search → Embedding Model: encodes query (evidence: `app/retrieve/semantic_search.py`)
- Hybrid Retrieval → Agentic Retrieval: feeds evidence (evidence: `README.md`)
- Agentic Retrieval → Grounded Answer: generates cited answer (evidence: `README.md`)
- FastAPI Application → Paper Routes: registers routes (evidence: `app/main.py`)
- FastAPI Application → PostgreSQL: creates tables (evidence: `app/main.py`)
- Paper Routes → PostgreSQL: reads and writes (evidence: `app/routes/papers.py`)
- Paper Routes → Paper Model: uses model (evidence: `app/routes/papers.py`)
- Paper Routes → Paper Schema: validates input (evidence: `app/routes/papers.py`)
