How perweena182/arxiv-ai-assistant works
Project type and purpose
• This is a Python-based, full-stack research assistant application.
• It loads and maintains up-to-date ArXiv metadata (via OAI-PMH + Sickle), embeds paper text, stores embeddings in Pinecone, builds a Neo4j knowledge graph, and serves a hybrid RAG/QA interface (via Azure OpenAI and Chainlit).
File structure
• Top-level scripts:
• metadata_pipeline.py – orchestrates OAI-PMH harvest and raw data ingestion
• metadata.py – metadata storage model and operations
• text_embedd.py – text embedding generation for papers
• knowledge_graph.py – builds and queries the Neo4j graph (ChatGroq integration)
• app.py – Chainlit app entrypoint serving the interactive UI
• Config: .chainlit/config.toml for Chainlit settings
• Data: unique_document_ids.csv for tracking processed records
• Requirements.txt lists dependencies (Sickle, Pinecone, neo4j, azure-ai, chainlit, etc.)
README insights
• Core pipeline: OAI-PMH → metadata repo
• RAG system: Pinecone + Azure OpenAI
• Hybrid QA: Pinecone vector search + Neo4j graph queries
• Frontend: Chainlit interactive chat interface
a. Main components
1. Data Ingestion Pipeline (metadata_pipeline.py + metadata.py)
2. Embedding Service (text_embedd.py)
3. Vector Store (Pinecone)
4. Knowledge Graph DB (Neo4j + knowledge_graph.py)
5. LLM Service (Azure OpenAI)
6. Interactive UI (Chainlit via app.py)
7. External ArXiv OAI-PMH endpoint
b. Relationships & interactions
• Pipeline pulls from ArXiv → metadata store
• Embedding Service reads metadata → generates embeddings → pushes to Pinecone
• Graph Builder reads metadata → constructs/updates Neo4j graph
• UI receives user query → simultaneously:
– invokes Pinecone vector search → returns top-k documents
– invokes Neo4j Cypher queries → returns graph insights
• UI sends combined context to Azure OpenAI → generates answer → responds to user
c. Architectural patterns
– Data Pipeline for ETL (metadata → stores)
– RAG (Retriever-Augmented Generation)
– Hybrid QA (vector + graph retrieval fusion)
– Modular, script-based micro-services (each Python script as a component)
d. Key technologies
– Python 3.x, Sickle (OAI-PMH), Pinecone SDK, Neo4j Python Driver, Azure OpenAI SDK, Chainlit
Detail capture
• Call out the dual-retriever pattern: vector vs. graph