How yohanes213/academic-research-assistant works
Project type and purpose
• This is a containerized, full-stack microservices application called “Academic Research Assistant.”
• Its goal is to scrape and index academic papers, store them in a vector database, and provide an AI-powered Q&A interface via a Streamlit frontend.
File structure
• Top-level directories each map to a Docker-ized service:
• airflow/ – workflow scheduler (periodic ingestion)
• paper_ingestion/ – scraper and ingestion API
• vector_db/ – vector storage API backed by Pinecone
• research_assistant/ – AI query service using Gemini, Tavily, LangSmith
• frontend/ – Streamlit web UI
• docker-compose.yml orchestrates all five containers plus shared network.
README for additional insights
• Ports:
• Frontend: 8501
• Paper Ingestion: 8000
• Vector DB: 5000
• Research Assistant: 5001
• Airflow: 8080
• External dependencies: PubMed (scraper), Pinecone, Google Gemini, Tavily, LangSmith.
a. Frontend UI
b. Paper Ingestion Service
c. Vector Database Service
d. Research Assistant Service
e. Airflow Scheduler
f. External systems: PubMed, Pinecone, Gemini API, Tavily API, LangSmith
Relationships and interactions
• User → Frontend (HTTP/Streamlit UI)
• Frontend → Research Assistant (POST /response)
• Frontend → Vector DB (GET /query) and Paper Ingestion (GET /scrape) for manual triggers
• Research Assistant → Vector DB (GET /query to fetch embeddings)
• Research Assistant → Gemini API, Tavily API, LangSmith for model inference & monitoring
• Paper Ingestion → PubMed (scrape), then → Vector DB (POST /upsert)
• Airflow → Paper Ingestion (GET /scrape) on schedule
Architectural patterns and design principles
• Microservices: each service is isolated in its own container and communicates over HTTP
• Layered architecture: UI layer (frontend), API layer (ingestion, vector_db, research_assistant), orchestration layer (Airflow), data/external layer (PubMed, Pinecone, LLMs)
• Event-driven ingestion pipeline scheduled by Airflow
• Use of external managed services for scalability (Pinecone, Gemini)
Technologies and libraries to annotate
• Streamlit (frontend)
• FastAPI + Uvicorn (paper_ingestion, vector_db, research_assistant)
• Apache Airflow (scheduler)
• Pinecone SDK, PubMed scraping logic
• Gemini (Google AI Studio), Tavily, LangSmith
• Research Assistant to Gemini, Tavily, LangSmith.
• Vector DB to Pinecone.