How jamwithai/arxiv-paper-curator works
Project Type and Purpose
• This is a full-stack, production-grade RAG (Retrieval-Augmented Generation) system called “arXiv Paper Curator.”
• Purpose: fetch papers from arXiv, parse PDFs, store metadata/content, index for search, and expose a FastAPI-based research assistant API.
File Structure
• Top level: Dockerfile, compose.yml, Makefile, src/, airflow/, notebooks/, static/, tests/.
• airflow/: Dockerfile, entrypoint, requirements, dags/ defining ETL pipelines (arxiv_ingestion, arxiv_paper_ingestion).
• notebooks/: Week-1 infra setup, Week-2 arXiv integration guides.
• compose.yml orchestrates FastAPI, PostgreSQL, OpenSearch, Airflow, Ollama.
README Insights
• Phases: Week 1 sets up infra (FastAPI, Postgres, OpenSearch, Airflow, Ollama).
• Week 2 builds data ingestion pipeline (ArxivClient, PDFParserService, MetadataFetcher, Airflow DAGs).
• Future: full RAG pipeline with vector search, LLM orchestration, observability.
• Emphasizes Docker Compose, Python 3.12+, UV package manager, testing.
Elements
• Airflow → MetadataFetcher → arXiv API → PDFParser → Postgres → OpenSearch → FastAPI → Ollama
Detail
• Major modules (routers, services, repositories, db interfaces)
• Both control flow (Airflow triggers) and data flow (paper content, metadata, search queries)
• Design patterns (e.g., “Asynchronous I/O”, “Dependency Injection”)