# yasout141516/arxiv-research-intelligence-pipeline architecture

> Architecture diagram of the GitHub repository yasout141516/arxiv-research-intelligence-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yasout141516/arxiv-research-intelligence-pipeline
- Repository: https://github.com/yasout141516/arxiv-research-intelligence-pipeline

## Overview

This repository is an arXiv research intelligence pipeline: it turns a paper snapshot into a validated analytical warehouse, charts, and a retrieval-augmented question-answering API. Its principal user workflow is to submit a filtered research question, retrieve relevant abstract chunks, and receive a grounded answer with paper citations. The graph also shows the separate paper-browsing endpoint and the documented ingestion, transformation, quality, visualization, and embedding stages. Some RAG implementation is excerpted only partially, so uncertain internal call wiring is omitted.

Main entry points: [`src/arxiv_pipeline/api.py`](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/api.py) (FastAPI Service), [`data/arxiv.db`](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/data/arxiv.db) (Analysis Tables), [`src/arxiv_pipeline/transform.py`](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/transform.py) (Warehouse Transform), [`src/arxiv_pipeline/visualize.py`](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/visualize.py) (Chart Generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Ingestion and Modeling"]
  node_ingest["Paper Ingestion<br/>[ingest.py]"]
  node_rawtable[("Raw Papers<br/>[arxiv.db]")]
  node_transform["Warehouse Transform<br/>[transform.py]"]
end

subgraph group_analysis["Warehouse and Analysis"]
  node_quality["Quality Checks<br/>[quality.py]"]
  node_warehouse[("Analysis Tables<br/>[arxiv.db]")]
  node_visualize["Chart Generation<br/>[visualize.py]"]
  node_charts["Analysis Charts"]
end

subgraph group_rag["Retrieval and Answers"]
  node_embedbuild["Embedding Build<br/>[rag.py]"]
  node_embedder["Text Embedder<br/>[rag.py]"]
  node_vectors[("Chroma Store<br/>[chroma.sqlite3]")]
  node_ragengine["RAG Answering<br/>[rag.py]"]
end

subgraph group_service["API and Shared Data"]
  node_api["FastAPI Service<br/>[api.py]"]
  node_dbhelpers["SQLite Access<br/>[db.py]"]
  node_config["Pipeline Config<br/>[config.py]"]
end

node_snapshot["arXiv Snapshot"]
node_apiuser(("API Client"))
node_groq{{"Groq Model API"}}

node_snapshot -->|"provides records"| node_ingest
node_ingest -->|"writes raw papers"| node_rawtable
node_transform -->|"reads raw papers"| node_rawtable
node_transform -->|"builds tables"| node_warehouse
node_quality -->|"checks tables"| node_warehouse
node_visualize -->|"reads aggregates"| node_warehouse
node_visualize -->|"writes charts"| node_charts
node_embedbuild -->|"reads abstracts"| node_warehouse
node_embedbuild -->|"requests embeddings"| node_embedder
node_embedbuild -->|"stores chunks"| node_vectors
node_apiuser -->|"sends requests"| node_api
node_api -->|"queries papers"| node_warehouse
node_api -->|"delegates questions"| node_ragengine
node_ragengine -->|"retrieves chunks"| node_vectors
node_ragengine -->|"requests grounded answer"| node_groq
node_api -->|"returns results"| node_apiuser
node_ingest -->|"reads settings"| node_config
node_transform -->|"uses SQLite"| node_dbhelpers
node_visualize -->|"uses SQLite"| node_dbhelpers
node_api -->|"uses SQLite"| node_dbhelpers

click node_ingest "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/ingest.py"
click node_rawtable "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/data/arxiv.db"
click node_transform "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/transform.py"
click node_quality "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/quality.py"
click node_warehouse "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/data/arxiv.db"
click node_visualize "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/visualize.py"
click node_charts "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/tree/main/plots"
click node_embedbuild "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py"
click node_embedder "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py"
click node_vectors "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/vector_store/chroma.sqlite3"
click node_api "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/api.py"
click node_ragengine "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py"
click node_dbhelpers "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/db.py"
click node_config "https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest,node_rawtable,node_transform,node_apiuser toneBlue
class node_quality,node_warehouse,node_visualize,node_charts toneAmber
class node_embedbuild,node_embedder,node_vectors,node_ragengine,node_groq toneMint
class node_api,node_dbhelpers,node_config toneRose
class node_snapshot toneIndigo
```

## Components

### Ingestion and Modeling

- [Paper Ingestion](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/ingest.py): `src/arxiv_pipeline/ingest.py`
- [Raw Papers](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/data/arxiv.db): `data/arxiv.db`
- [Warehouse Transform](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/transform.py): `src/arxiv_pipeline/transform.py`

### Warehouse and Analysis

- [Quality Checks](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/quality.py): `src/arxiv_pipeline/quality.py`
- [Analysis Tables](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/data/arxiv.db): `data/arxiv.db`
- [Chart Generation](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/visualize.py): `src/arxiv_pipeline/visualize.py`
- [Analysis Charts](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/tree/main/plots): `plots`

### Retrieval and Answers

- [Embedding Build](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py): `src/arxiv_pipeline/rag.py`
- [Text Embedder](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py): `src/arxiv_pipeline/rag.py`
- [Chroma Store](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/vector_store/chroma.sqlite3): `vector_store/chroma.sqlite3`
- [RAG Answering](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/rag.py): `src/arxiv_pipeline/rag.py`

### API and Shared Data

- [FastAPI Service](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/api.py): `src/arxiv_pipeline/api.py`
- [SQLite Access](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/db.py): `src/arxiv_pipeline/db.py`
- [Pipeline Config](https://github.com/yasout141516/arxiv-research-intelligence-pipeline/blob/main/src/arxiv_pipeline/config.py): `src/arxiv_pipeline/config.py`

### Other components

- **arXiv Snapshot**
- **API Client**
- **Groq Model API**

## Connections

- arXiv Snapshot → Paper Ingestion: provides records
- Paper Ingestion → Raw Papers: writes raw papers
- Warehouse Transform → Raw Papers: reads raw papers
- Warehouse Transform → Analysis Tables: builds tables
- Quality Checks → Analysis Tables: checks tables
- Chart Generation → Analysis Tables: reads aggregates
- Chart Generation → Analysis Charts: writes charts
- Embedding Build → Analysis Tables: reads abstracts
- Embedding Build → Text Embedder: requests embeddings
- Embedding Build → Chroma Store: stores chunks
- API Client → FastAPI Service: sends requests
- FastAPI Service → Analysis Tables: queries papers
- FastAPI Service → RAG Answering: delegates questions
- RAG Answering → Chroma Store: retrieves chunks
- RAG Answering → Groq Model API: requests grounded answer
- FastAPI Service → API Client: returns results
- Paper Ingestion → Pipeline Config: reads settings
- Warehouse Transform → SQLite Access: uses SQLite
- Chart Generation → SQLite Access: uses SQLite
- FastAPI Service → SQLite Access: uses SQLite