# sachin210506/scholar architecture

> Architecture diagram of the GitHub repository sachin210506/scholar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sachin210506/scholar
- Repository: https://github.com/sachin210506/scholar

## Overview

#### 1. Purpose and Runtime Entry Points
scholAR is a local-first academic document retrieval and RAG system: users upload PDFs, search indexed course material, receive cited answers, generate practice questions, and retain chat history. The React/Vite client starts at `frontend/src/main.jsx` and renders the primary interface in `frontend/src/App.jsx`. The FastAPI service entry point is `backend/src/api.py`, run as `uvicorn backend.src.api:app`.

#### 2. API Orchestration Boundary
`backend/src/api.py` is the application boundary between browser workflows and backend services. It exposes health/stats, PDF upload, hybrid search, cited answering, practice-question generation, and chat-session routes. The frontend is a single main UI surface that calls this API rather than directly accessing indexes, files, LLMs, or PostgreSQL.
- `backend/src/api.py`
- `frontend/src/App.jsx`
- `frontend/src/main.jsx`

#### 3. Document Ingestion and Local Corpus
Upload processing persists PDFs locally, derives file-hash-based document identity, captures academic metadata, extracts page-level text, and produces retrieval chunks. Extraction is Markdown/page aware and falls back to Tesseract OCR for low-text scanned pages. Chunking preserves page provenance and can merge adjacent continued topics; stable chunk IDs permit both indexes to refer to the same evidence.
- `backend/src/extract.py`
- `backend/src/chunking.py`
- `backend/src/embed_index.py`

#### 4. Dual Retrieval Indexes and Ranking
Indexed chunks are embedded with local `sentence-transformers/all-MiniLM-L6-v2` and stored in ChromaDB, while a persisted custom BM25 index supplies lexical matching. `search.py` joins these result sets by stable chunk ID and applies hybrid scoring: semantic similarity, BM25, exact terms, headings, and source credibility. Request-level metadata filters constrain subject, document/source type, and credibility. A retrieval-confidence gate prevents downstream generation from treating weak context as evidence.
- `backend/src/embed_index.py`
- `backend/src/bm25_index.py`
- `backend/src/search.py`

#### 5. Grounded Generation and Credibility
Answer and practice workflows consume retrieved chunks, build citation-aware prompts, and refuse or request clarification when retrieval is weak. Generated answer/question credibility is separately assessed from source credibility and relevance, using retrieval strength, source quality/variety, and citation use. This is a deliberate boundary: trusted documents do not override poor query relevance.
- `backend/src/rag_answer.py`
- `backend/src/generate_questions.py`
- `backend/src/output_credibility.py`

#### 6. LLM Provider Boundary
Generation routes use an abstraction that invokes Gemini as the configured primary model and Groq as fallback for provider failures such as timeout, quota, or credentials. Provider keys, selected models, and database connection details are supplied through environment variables rather than application code.
- `backend/src/llm_provider.py`
- `.env.example`

#### 7. Stateful Chat and External Persistence
Chat has a separate persistence path from corpus data: sessions/messages are stored in Neon PostgreSQL through the database and store layers. Before retrieval, follow-up prompts are rewritten into standalone search queries using a recent-message window; guardrails reject rewrites that lose the prior topic. The final response can still use original wording and conversation context.
- `backend/src/db.py`
- `backend/src/chat_store.py`
- `backend/src/contextual_query.py`

#### 8. Storage and Infrastructure Split
The system intentionally distributes state by workload: local filesystem for uploaded PDFs, local ChromaDB for vectors/chunks, local persisted BM25 for lexical search, and Neon PostgreSQL for durable chat. Material external dependencies are Gemini, Groq, Neon/PostgreSQL, and locally installed Tesseract for OCR; FastAPI/Pydantic serve the backend and React/Vite serve the browser client.

Main entry points: [`backend/src/api.py`](https://github.com/sachin210506/scholar/blob/main/backend/src/api.py) (FastAPI service), [`backend/src/search.py`](https://github.com/sachin210506/scholar/blob/main/backend/src/search.py) (Hybrid search), [`backend/src/extract.py`](https://github.com/sachin210506/scholar/blob/main/backend/src/extract.py) (PDF extraction), [`backend/src/rag_answer.py`](https://github.com/sachin210506/scholar/blob/main/backend/src/rag_answer.py) (Cited answers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Browser client"]
  node_vite["React/Vite bootstrap<br/>client entry<br/>[main.jsx]"]
  node_ui["Main UI<br/>React interface<br/>[App.jsx]"]
end

subgraph group_api["API boundary"]
  node_api_server["FastAPI service<br/>API orchestration<br/>[api.py]"]
end

subgraph group_corpus["Corpus ingestion"]
  node_extract["PDF extraction<br/>page text extractor<br/>[extract.py]"]
  node_pdf_store[("Local PDF corpus<br/>filesystem storage")]
  node_chunking["Provenance chunking<br/>chunk builder<br/>[chunking.py]"]
end

subgraph group_retrieval["Hybrid retrieval"]
  node_embedding["Embedding indexer<br/>vector index writer<br/>[embed_index.py]"]
  node_chroma[("ChromaDB<br/>local vector store")]
  node_bm25["BM25 index<br/>lexical index<br/>[bm25_index.py]"]
  node_search["Hybrid search<br/>retrieval ranker<br/>[search.py]"]
end

subgraph group_generation["Grounded generation"]
  node_answer["Cited answers<br/>RAG workflow<br/>[rag_answer.py]"]
  node_questions["Practice questions<br/>generation workflow"]
  node_credibility["Output credibility<br/>generation assessor"]
  node_llm{{"LLM provider<br/>provider abstraction<br/>[llm_provider.py]"}}
end

subgraph group_chat["Chat persistence"]
  node_contextual_query["Contextual query rewrite<br/>chat query resolver"]
  node_chat_store["Chat store<br/>session repository<br/>[chat_store.py]"]
  node_db["Database layer<br/>PostgreSQL access<br/>[db.py]"]
  node_postgres[("Neon PostgreSQL<br/>durable chat database")]
end

node_providers{{"Gemini / Groq<br/>external LLMs"}}
node_tesseract{{"Tesseract<br/>local OCR dependency"}}

node_vite -->|"renders"| node_ui
node_ui -->|"API requests"| node_api_server
node_api_server -->|"stores uploads"| node_pdf_store
node_api_server -->|"starts ingestion"| node_extract
node_pdf_store -->|"PDF input"| node_extract
node_extract -.->|"scanned-page OCR"| node_tesseract
node_extract -->|"page text + metadata"| node_chunking
node_chunking -->|"stable chunks"| node_embedding
node_chunking -->|"stable chunks"| node_bm25
node_embedding -->|"vectors + evidence"| node_chroma
node_api_server -->|"search requests"| node_search
node_chroma -->|"semantic candidates"| node_search
node_bm25 -->|"lexical candidates"| node_search
node_api_server -->|"answer route"| node_answer
node_api_server -->|"practice route"| node_questions
node_search -->|"gated evidence"| node_answer
node_search -->|"gated evidence"| node_questions
node_answer -->|"answer assessment"| node_credibility
node_questions -->|"question assessment"| node_credibility
node_answer -->|"grounded prompt"| node_llm
node_questions -->|"grounded prompt"| node_llm
node_llm -->|"primary / fallback"| node_providers
node_api_server -->|"follow-up prompt"| node_contextual_query
node_chat_store -->|"recent messages"| node_contextual_query
node_contextual_query -->|"standalone query"| node_search
node_api_server -->|"read/write chat"| node_chat_store
node_chat_store -->|"sessions + messages"| node_db
node_db -->|"durable persistence"| node_postgres

click node_vite "https://github.com/sachin210506/scholar/blob/main/frontend/src/main.jsx"
click node_ui "https://github.com/sachin210506/scholar/blob/main/frontend/src/App.jsx"
click node_api_server "https://github.com/sachin210506/scholar/blob/main/backend/src/api.py"
click node_extract "https://github.com/sachin210506/scholar/blob/main/backend/src/extract.py"
click node_chunking "https://github.com/sachin210506/scholar/blob/main/backend/src/chunking.py"
click node_embedding "https://github.com/sachin210506/scholar/blob/main/backend/src/embed_index.py"
click node_bm25 "https://github.com/sachin210506/scholar/blob/main/backend/src/bm25_index.py"
click node_search "https://github.com/sachin210506/scholar/blob/main/backend/src/search.py"
click node_answer "https://github.com/sachin210506/scholar/blob/main/backend/src/rag_answer.py"
click node_questions "https://github.com/sachin210506/scholar/blob/main/backend/src/generate_questions.py"
click node_credibility "https://github.com/sachin210506/scholar/blob/main/backend/src/output_credibility.py"
click node_llm "https://github.com/sachin210506/scholar/blob/main/backend/src/llm_provider.py"
click node_contextual_query "https://github.com/sachin210506/scholar/blob/main/backend/src/contextual_query.py"
click node_chat_store "https://github.com/sachin210506/scholar/blob/main/backend/src/chat_store.py"
click node_db "https://github.com/sachin210506/scholar/blob/main/backend/src/db.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_vite,node_ui toneBlue
class node_api_server toneAmber
class node_extract,node_pdf_store,node_chunking toneMint
class node_embedding,node_chroma,node_bm25,node_search toneRose
class node_answer,node_questions,node_credibility,node_llm toneIndigo
class node_contextual_query,node_chat_store,node_db,node_postgres toneTeal
class node_providers,node_tesseract toneNeutral
```

## Components

### Browser client

- [React/Vite bootstrap](https://github.com/sachin210506/scholar/blob/main/frontend/src/main.jsx): client entry · `frontend/src/main.jsx`
- [Main UI](https://github.com/sachin210506/scholar/blob/main/frontend/src/App.jsx): React interface · Uploads, search, answers, practice, and chat. · `frontend/src/App.jsx`

### API boundary

- [FastAPI service](https://github.com/sachin210506/scholar/blob/main/backend/src/api.py): API orchestration · Owns browser-facing upload, retrieval, generation, and chat routes. · `backend/src/api.py`

### Corpus ingestion

- [PDF extraction](https://github.com/sachin210506/scholar/blob/main/backend/src/extract.py): page text extractor · Uses Tesseract OCR for low-text scanned pages. · `backend/src/extract.py`
- **Local PDF corpus**: filesystem storage · Stores uploaded PDFs under hash-based document identities.
- [Provenance chunking](https://github.com/sachin210506/scholar/blob/main/backend/src/chunking.py): chunk builder · Creates stable, page-aware chunk IDs. · `backend/src/chunking.py`

### Hybrid retrieval

- [Embedding indexer](https://github.com/sachin210506/scholar/blob/main/backend/src/embed_index.py): vector index writer · Embeds chunks with local all-MiniLM-L6-v2. · `backend/src/embed_index.py`
- **ChromaDB**: local vector store · Persists vectors and chunk evidence.
- [BM25 index](https://github.com/sachin210506/scholar/blob/main/backend/src/bm25_index.py): lexical index · Persists lexical matches keyed by stable chunk ID. · `backend/src/bm25_index.py`
- [Hybrid search](https://github.com/sachin210506/scholar/blob/main/backend/src/search.py): retrieval ranker · Joins vector and BM25 evidence, filters metadata, and gates weak retrieval. · `backend/src/search.py`

### Grounded generation

- [Cited answers](https://github.com/sachin210506/scholar/blob/main/backend/src/rag_answer.py): RAG workflow · Builds citation-aware answers or declines weak evidence. · `backend/src/rag_answer.py`
- [Practice questions](https://github.com/sachin210506/scholar/blob/main/backend/src/generate_questions.py): generation workflow · Generates questions only from retrieved context. · `backend/src/generate_questions.py`
- [Output credibility](https://github.com/sachin210506/scholar/blob/main/backend/src/output_credibility.py): generation assessor · Separately evaluates relevance, citations, and source variety. · `backend/src/output_credibility.py`
- [LLM provider](https://github.com/sachin210506/scholar/blob/main/backend/src/llm_provider.py): provider abstraction · Uses Gemini primarily and Groq on provider failure. · `backend/src/llm_provider.py`

### Chat persistence

- [Contextual query rewrite](https://github.com/sachin210506/scholar/blob/main/backend/src/contextual_query.py): chat query resolver · Rewrites follow-ups with topic-loss guardrails. · `backend/src/contextual_query.py`
- [Chat store](https://github.com/sachin210506/scholar/blob/main/backend/src/chat_store.py): session repository · `backend/src/chat_store.py`
- [Database layer](https://github.com/sachin210506/scholar/blob/main/backend/src/db.py): PostgreSQL access · `backend/src/db.py`
- **Neon PostgreSQL**: durable chat database · Stores chat sessions and messages.

### Other components

- **Gemini / Groq**: external LLMs
- **Tesseract**: local OCR dependency

## Connections

- React/Vite bootstrap → Main UI: renders
- Main UI → FastAPI service: API requests
- FastAPI service → Local PDF corpus: stores uploads
- FastAPI service → PDF extraction: starts ingestion
- Local PDF corpus → PDF extraction: PDF input
- PDF extraction → Tesseract: scanned-page OCR
- PDF extraction → Provenance chunking: page text + metadata
- Provenance chunking → Embedding indexer: stable chunks
- Provenance chunking → BM25 index: stable chunks
- Embedding indexer → ChromaDB: vectors + evidence
- FastAPI service → Hybrid search: search requests
- ChromaDB → Hybrid search: semantic candidates
- BM25 index → Hybrid search: lexical candidates
- FastAPI service → Cited answers: answer route
- FastAPI service → Practice questions: practice route
- Hybrid search → Cited answers: gated evidence
- Hybrid search → Practice questions: gated evidence
- Cited answers → Output credibility: answer assessment
- Practice questions → Output credibility: question assessment
- Cited answers → LLM provider: grounded prompt
- Practice questions → LLM provider: grounded prompt
- LLM provider → Gemini / Groq: primary / fallback
- FastAPI service → Contextual query rewrite: follow-up prompt
- Chat store → Contextual query rewrite: recent messages
- Contextual query rewrite → Hybrid search: standalone query
- FastAPI service → Chat store: read/write chat
- Chat store → Database layer: sessions + messages
- Database layer → Neon PostgreSQL: durable persistence