How aryanprabhat-spec/mit-gpt works
MIT GPT is a course-notes RAG application: users ask questions and receive context-grounded answers with source metadata. Its runtime loads PDF/PPTX notes, creates metadata-preserving chunks, and builds or reloads sparse and dense indexes. Questions are expanded and searched through hybrid retrieval, reranked, then sent with retrieved context to an LLM; results can be shown interactively or exported for evaluation. The README documents the major stages; sampled source confirms pipeline orchestration and persistence. Hybrid-retriever internals were not sampled, so those documented retrieval links cite the README.
Main entry points: pipeline.py RAG Pipeline · hybrid.py Hybrid Retrieval · export_rag_outputs.py Batch Export · generator.py Grounded Generation
Components 18
User Interfaces
- Interactive Q&A
main.py - Batch Export
export_rag_outputs.py - RAG Pipeline
src/pipeline.py - RAG Results JSON
outputs
Document Indexing
- Course Notes
data/raw - PDF/PPTX Loader
src/ingestion/loader.py - Semantic Chunking
src/processing/chunker.py - Persisted Chunks
src/processing/chunker.py - Configuration
src/config/settings.py
Retrieval
- BM25 Retriever
src/retrieval/bm25.py - Dense Vector Store
src/retrieval/vector_store.py - Hybrid Retrieval
src/retrieval/hybrid.py - Query Expansion
src/retrieval/multi_query.py - Cross-Encoder Reranker
src/retrieval/reranker.py
Answer Generation
- Grounded Generation
src/generation/generator.py
Other components
- User
- Groq LLM
- Evaluation Consumer
Connections 25
- User → Interactive Q&A: asks question
README.md - Interactive Q&A → RAG Pipeline: builds pipeline
main.py - User → Batch Export: starts export
README.md - Batch Export → RAG Pipeline: builds pipeline
export_rag_outputs.py - Course Notes → PDF/PPTX Loader: loads files
src/ingestion/loader.py - RAG Pipeline → PDF/PPTX Loader: loads documents
src/pipeline.py - RAG Pipeline → Semantic Chunking: chunks documents
src/pipeline.py - Semantic Chunking → Persisted Chunks: saves and loads
src/processing/chunker.py - RAG Pipeline → Persisted Chunks: persists or restores
src/pipeline.py - RAG Pipeline → BM25 Retriever: builds retriever
src/pipeline.py - RAG Pipeline → Dense Vector Store: builds or loads
src/pipeline.py - BM25 Retriever → Hybrid Retrieval: sparse candidates
README.md - Dense Vector Store → Hybrid Retrieval: dense candidates
README.md - RAG Pipeline → Hybrid Retrieval: assembles retrieval
src/pipeline.py - Hybrid Retrieval → Query Expansion: retrieves variants
README.md - RAG Pipeline → Query Expansion: creates retriever
src/pipeline.py - RAG Pipeline → Cross-Encoder Reranker: reranks candidates
src/pipeline.py - Query Expansion → Cross-Encoder Reranker: returns candidates
src/pipeline.py - RAG Pipeline → Grounded Generation: builds context and answer
src/pipeline.py - Grounded Generation → Groq LLM: requests completion
src/generation/generator.py - RAG Pipeline → Configuration: reads index path
src/pipeline.py - Grounded Generation → Configuration: reads model settings
src/generation/generator.py - RAG Pipeline → Interactive Q&A: returns answer and sources
main.py - Batch Export → RAG Results JSON: writes JSON
export_rag_outputs.py - RAG Results JSON → Evaluation Consumer: supplies evaluation input
README.md