sachin210506/scholar

1. Purpose and Runtime Entry Points

scholAR is a local-first academic document retrieval and RAG system: users upload PDFs, search indexed course material, receive cited answers, generate practice questions, and retain chat history. The React/Vite client starts at frontend/src/main.jsx and renders the primary interface in frontend/src/App.jsx. The FastAPI service entry point is backend/src/api.py, run as uvicorn backend.src.api:app.

2. API Orchestration Boundary

backend/src/api.py is the application boundary between browser workflows and backend services. It exposes health/stats, PDF upload, hybrid search, cited answering, practice-question generation, and chat-session routes. The frontend is a single main UI surface that calls this API rather than directly accessing indexes, files, LLMs, or PostgreSQL.

• backend/src/api.py

• frontend/src/App.jsx

• frontend/src/main.jsx

3. Document Ingestion and Local Corpus

Upload processing persists PDFs locally, derives file-hash-based document identity, captures academic metadata, extracts page-level text, and produces retrieval chunks. Extraction is Markdown/page aware and falls back to Tesseract OCR for low-text scanned pages. Chunking preserves page provenance and can merge adjacent continued topics; stable chunk IDs permit both indexes to refer to the same evidence.

• backend/src/extract.py

• backend/src/chunking.py

• backend/src/embed_index.py

4. Dual Retrieval Indexes and Ranking

Indexed chunks are embedded with local sentence-transformers/all-MiniLM-L6-v2 and stored in ChromaDB, while a persisted custom BM25 index supplies lexical matching. search.py joins these result sets by stable chunk ID and applies hybrid scoring: semantic similarity, BM25, exact terms, headings, and source credibility. Request-level metadata filters constrain subject, document/source type, and credibility. A retrieval-confidence gate prevents downstream generation from treating weak context as evidence.

• backend/src/embed_index.py

• backend/src/bm25_index.py

• backend/src/search.py

5. Grounded Generation and Credibility

Answer and practice workflows consume retrieved chunks, build citation-aware prompts, and refuse or request clarification when retrieval is weak. Generated answer/question credibility is separately assessed from source credibility and relevance, using retrieval strength, source quality/variety, and citation use. This is a deliberate boundary: trusted documents do not override poor query relevance.

• backend/src/rag_answer.py

• backend/src/generate_questions.py

• backend/src/output_credibility.py

6. LLM Provider Boundary

Generation routes use an abstraction that invokes Gemini as the configured primary model and Groq as fallback for provider failures such as timeout, quota, or credentials. Provider keys, selected models, and database connection details are supplied through environment variables rather than application code.

• backend/src/llm_provider.py

• .env.example

7. Stateful Chat and External Persistence

Chat has a separate persistence path from corpus data: sessions/messages are stored in Neon PostgreSQL through the database and store layers. Before retrieval, follow-up prompts are rewritten into standalone search queries using a recent-message window; guardrails reject rewrites that lose the prior topic. The final response can still use original wording and conversation context.

• backend/src/db.py

• backend/src/chat_store.py

• backend/src/contextual_query.py

8. Storage and Infrastructure Split

The system intentionally distributes state by workload: local filesystem for uploaded PDFs, local ChromaDB for vectors/chunks, local persisted BM25 for lexical search, and Neon PostgreSQL for durable chat. Material external dependencies are Gemini, Groq, Neon/PostgreSQL, and locally installed Tesseract for OCR; FastAPI/Pydantic serve the backend and React/Vite serve the browser client.