How pranav1011/research_paper_assistant works
Project Type and Purpose
• This is a Python-based Research Paper Assistant combining a web interface (Gradio) and a CLI pipeline.
• It ingests PDF research papers, parses and chunks them, builds a FAISS vector index, retrieves relevant passages, and uses an LLM to answer user questions.
a. Presentation Layer
• Web UI (app.py – Gradio server on localhost:7860)
• CLI Orchestrator (automated_pipeline.py / main.py)
b. Business Logic / Orchestration
• Orchestrator Module (automated_pipeline.py / main.py)
• Configuration Module (config/config.py)
• Backend Prompt Templates (backend/prompts/templates.py)
c. Data Processing Services
• PDF Parser (scripts/prepare_magic_pdf_chunks.py or a mineru-based parser)
• Chunker (scripts/chunking_md_sections.py / scripts/process_chunks.py)
• Embedding & Index Builder (vectorstore/faiss_index)
• Hybrid Retriever (vectorstore/hybrid_retriever.py)
d. External Services
• LLM API Client (scripts/llm_answer.py calling OpenAI/HuggingFace)
• Evaluation Scripts (scripts/llm_eval_metric.py, scripts/llm_evaluate_answers.py)
e. Data Stores
• Vector Store (FAISS indices on disk under vectorstore/faiss_index)
• Raw & Intermediate Data (data/papers, data/…chunks)
f. Support Modules
• Logging & Config (requirements.txt, config/)
• Tests (tests/…)
1. User → Web UI or CLI
2. UI/CLI → Orchestrator
3. Orchestrator
→ PDF Parser → Extracted Text → Chunker → Text Chunks
→ Embedding Service → Vectorstore Index (persisted on disk)
4. User Query → Hybrid Retriever → Candidate Chunks
5. Candidate Chunks + Prompt Templates → LLM API Client → Model Response
6. Response → UI/CLI → User
7. (Optional) Orchestrator → Evaluation Scripts → Metrics → Logs
Architectural Patterns & Design Principles
• Layered Architecture: Presentation (UI/CLI) → Orchestration → Services → Storage → External
• Modular Services: separate scripts for parsing, chunking, retrieval, evaluation
• Config-Driven: config/config.py centralizes environment settings
• Reusable Prompt Templates: backend/prompts/templates.py for LLM queries
• Test-Driven: tests/ directory covers each major component