# perweena182/arxiv-ai-assistant architecture

> Architecture diagram of the GitHub repository perweena182/arxiv-ai-assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/perweena182/arxiv-ai-assistant
- Repository: https://github.com/perweena182/arxiv-ai-assistant

## Overview

##### Project type and purpose
- This is a Python-based, full-stack research assistant application.
- It loads and maintains up-to-date ArXiv metadata (via OAI-PMH + Sickle), embeds paper text, stores embeddings in Pinecone, builds a Neo4j knowledge graph, and serves a hybrid RAG/QA interface (via Azure OpenAI and Chainlit).

##### File structure
- Top-level scripts:
- metadata_pipeline.py – orchestrates OAI-PMH harvest and raw data ingestion
- metadata.py – metadata storage model and operations
- text_embedd.py – text embedding generation for papers
- knowledge_graph.py – builds and queries the Neo4j graph (ChatGroq integration)
- app.py – Chainlit app entrypoint serving the interactive UI
- Config: .chainlit/config.toml for Chainlit settings
- Data: unique_document_ids.csv for tracking processed records
- Requirements.txt lists dependencies (Sickle, Pinecone, neo4j, azure-ai, chainlit, etc.)

##### README insights
- Core pipeline: OAI-PMH → metadata repo
- RAG system: Pinecone + Azure OpenAI
- Hybrid QA: Pinecone vector search + Neo4j graph queries
- Frontend: Chainlit interactive chat interface

a. Main components
1. Data Ingestion Pipeline (metadata_pipeline.py + metadata.py)
2. Embedding Service (text_embedd.py)
3. Vector Store (Pinecone)
4. Knowledge Graph DB (Neo4j + knowledge_graph.py)
5. LLM Service (Azure OpenAI)
6. Interactive UI (Chainlit via app.py)
7. External ArXiv OAI-PMH endpoint
b. Relationships & interactions
- Pipeline pulls from ArXiv → metadata store
- Embedding Service reads metadata → generates embeddings → pushes to Pinecone
- Graph Builder reads metadata → constructs/updates Neo4j graph
- UI receives user query → simultaneously:
– invokes Pinecone vector search → returns top-k documents
– invokes Neo4j Cypher queries → returns graph insights
- UI sends combined context to Azure OpenAI → generates answer → responds to user
c. Architectural patterns
– Data Pipeline for ETL (metadata → stores)
– RAG (Retriever-Augmented Generation)
– Hybrid QA (vector + graph retrieval fusion)
– Modular, script-based micro-services (each Python script as a component)
d. Key technologies
– Python 3.x, Sickle (OAI-PMH), Pinecone SDK, Neo4j Python Driver, Azure OpenAI SDK, Chainlit

##### Detail capture
- Call out the dual-retriever pattern: vector vs. graph

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External source
    subgraph "External Sources"
        ArXiv["ArXiv OAI-PMH Endpoint"]:::external
    end

    %% Data Ingestion
    subgraph "Data Ingestion Pipeline"
        MP["metadata_pipeline.py"]:::ingestion
        MD["metadata.py"]:::ingestion
        IDs["unique_document_ids.csv"]:::config
        MP -->|"harvest via Sickle"| MD
        MP -->|"track processed IDs"| IDs
        ArXiv -->|"OAI-PMH fetch"| MP
    end

    %% Embedding Service
    subgraph "Embedding Service"
        TE["text_embedd.py"]:::ingestion
        MD -->|"read metadata"| TE
        TE -->|"upsert embeddings"| Pinecone["Pinecone Vector Store"]:::datastore
    end

    %% Knowledge Graph
    subgraph "Knowledge Graph Builder"
        KG["knowledge_graph.py"]:::ingestion
        MD -->|"read metadata"| KG
        KG -->|"build/update graph"| Neo4j["Neo4j DB"]:::datastore
    end

    %% Configuration & Docs
    subgraph "Configuration & Documentation"
        Rq["requirements.txt"]:::config
        RM["README.md"]:::config
        CM["chainlit.md"]:::config
    end

    %% Interactive UI
    subgraph "Interactive UI (Chainlit)"
        UI["app.py"]:::ui
        Cfg[".chainlit/config.toml"]:::config
        UI --> Cfg
        UI --> UX["Chainlit Chat Interface"]:::ui
    end

    %% Retrieval & LLM
    UX -->|"vector search\n(cosine similarity)"| Pinecone
    UX -->|"graph query\n(Cypher)"| Neo4j
    UX -->|"send combined context"| LLM["Azure OpenAI"]:::ai
    LLM -->|"answer generation"| UX

    %% Click Events
    click MP "https://github.com/perweena182/arxiv-ai-assistant/blob/main/metadata_pipeline.py"
    click MD "https://github.com/perweena182/arxiv-ai-assistant/blob/main/metadata.py"
    click TE "https://github.com/perweena182/arxiv-ai-assistant/blob/main/text_embedd.py"
    click KG "https://github.com/perweena182/arxiv-ai-assistant/blob/main/knowledge_graph.py"
    click UI "https://github.com/perweena182/arxiv-ai-assistant/blob/main/app.py"
    click Cfg "https://github.com/perweena182/arxiv-ai-assistant/blob/main/.chainlit/config.toml"
    click Rq "https://github.com/perweena182/arxiv-ai-assistant/blob/main/requirements.txt"
    click IDs "https://github.com/perweena182/arxiv-ai-assistant/blob/main/unique_document_ids.csv"
    click RM "https://github.com/perweena182/arxiv-ai-assistant/blob/main/README.md"
    click CM "https://github.com/perweena182/arxiv-ai-assistant/blob/main/chainlit.md"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#3399ff
    classDef datastore fill:#d4edda,stroke:#28a745
    classDef ai fill:#ffe5b4,stroke:#ff9933
    classDef ui fill:#e6ccff,stroke:#9933ff
    classDef config fill:#f0f0f0,stroke:#666666,stroke-dasharray: 2 2
    classDef external fill:#f5f5f5,stroke:#999999,stroke-dasharray: 2 2
```
