How yasout141516/arxiv-research-intelligence-pipeline works
This repository is an arXiv research intelligence pipeline: it turns a paper snapshot into a validated analytical warehouse, charts, and a retrieval-augmented question-answering API. Its principal user workflow is to submit a filtered research question, retrieve relevant abstract chunks, and receive a grounded answer with paper citations. The graph also shows the separate paper-browsing endpoint and the documented ingestion, transformation, quality, visualization, and embedding stages. Some RAG implementation is excerpted only partially, so uncertain internal call wiring is omitted.
Main entry points: api.py FastAPI Service · arxiv.db Analysis Tables · transform.py Warehouse Transform · visualize.py Chart Generation
Components 17
Ingestion and Modeling
- Paper Ingestion
src/arxiv_pipeline/ingest.py - Raw Papers
data/arxiv.db - Warehouse Transform
src/arxiv_pipeline/transform.py
Warehouse and Analysis
- Quality Checks
src/arxiv_pipeline/quality.py - Analysis Tables
data/arxiv.db - Chart Generation
src/arxiv_pipeline/visualize.py - Analysis Charts
plots
Retrieval and Answers
- Embedding Build
src/arxiv_pipeline/rag.py - Text Embedder
src/arxiv_pipeline/rag.py - Chroma Store
vector_store/chroma.sqlite3 - RAG Answering
src/arxiv_pipeline/rag.py
API and Shared Data
- FastAPI Service
src/arxiv_pipeline/api.py - SQLite Access
src/arxiv_pipeline/db.py - Pipeline Config
src/arxiv_pipeline/config.py
Other components
- arXiv Snapshot
- API Client
- Groq Model API
Connections 20
- arXiv Snapshot → Paper Ingestion: provides records
- Paper Ingestion → Raw Papers: writes raw papers
- Warehouse Transform → Raw Papers: reads raw papers
- Warehouse Transform → Analysis Tables: builds tables
- Quality Checks → Analysis Tables: checks tables
- Chart Generation → Analysis Tables: reads aggregates
- Chart Generation → Analysis Charts: writes charts
- Embedding Build → Analysis Tables: reads abstracts
- Embedding Build → Text Embedder: requests embeddings
- Embedding Build → Chroma Store: stores chunks
- API Client → FastAPI Service: sends requests
- FastAPI Service → Analysis Tables: queries papers
- FastAPI Service → RAG Answering: delegates questions
- RAG Answering → Chroma Store: retrieves chunks
- RAG Answering → Groq Model API: requests grounded answer
- FastAPI Service → API Client: returns results
- Paper Ingestion → Pipeline Config: reads settings
- Warehouse Transform → SQLite Access: uses SQLite
- Chart Generation → SQLite Access: uses SQLite
- FastAPI Service → SQLite Access: uses SQLite