# aryanprabhat-spec/mit-gpt architecture

> Architecture diagram of the GitHub repository aryanprabhat-spec/mit-gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aryanprabhat-spec/mit-gpt
- Repository: https://github.com/aryanprabhat-spec/mit-gpt

## Overview

MIT GPT is a course-notes RAG application: users ask questions and receive context-grounded answers with source metadata. Its runtime loads PDF/PPTX notes, creates metadata-preserving chunks, and builds or reloads sparse and dense indexes. Questions are expanded and searched through hybrid retrieval, reranked, then sent with retrieved context to an LLM; results can be shown interactively or exported for evaluation. The README documents the major stages; sampled source confirms pipeline orchestration and persistence. Hybrid-retriever internals were not sampled, so those documented retrieval links cite the README.

Main entry points: [`src/pipeline.py`](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/pipeline.py) (RAG Pipeline), [`src/retrieval/hybrid.py`](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/hybrid.py) (Hybrid Retrieval), [`export_rag_outputs.py`](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/export_rag_outputs.py) (Batch Export), [`src/generation/generator.py`](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/generation/generator.py) (Grounded Generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["User Interfaces"]
  node_main["Interactive Q&amp;A<br/>[main.py]"]
  node_batch["Batch Export"]
  node_pipeline["RAG Pipeline<br/>[pipeline.py]"]
  node_results["RAG Results JSON"]
end

subgraph group_indexing["Document Indexing"]
  node_source_docs["Course Notes"]
  node_loader["PDF/PPTX Loader<br/>[loader.py]"]
  node_chunker["Semantic Chunking<br/>[chunker.py]"]
  node_chunk_store[("Persisted Chunks<br/>[chunker.py]")]
  node_settings["Configuration<br/>[settings.py]"]
end

subgraph group_retrieval["Retrieval"]
  node_bm25["BM25 Retriever<br/>[bm25.py]"]
  node_vector[("Dense Vector Store<br/>[vector_store.py]")]
  node_hybrid["Hybrid Retrieval<br/>[hybrid.py]"]
  node_multi_query["Query Expansion<br/>[multi_query.py]"]
  node_reranker["Cross-Encoder Reranker<br/>[reranker.py]"]
end

subgraph group_answering["Answer Generation"]
  node_generator["Grounded Generation<br/>[generator.py]"]
end

node_user(("User"))
node_groq{{"Groq LLM"}}
node_evaluator(("Evaluation Consumer"))

node_user -->|"asks question"| node_main
node_main -->|"builds pipeline"| node_pipeline
node_user -->|"starts export"| node_batch
node_batch -->|"builds pipeline"| node_pipeline
node_source_docs -->|"loads files"| node_loader
node_pipeline -->|"loads documents"| node_loader
node_pipeline -->|"chunks documents"| node_chunker
node_chunker -->|"saves and loads"| node_chunk_store
node_pipeline -->|"persists or restores"| node_chunk_store
node_pipeline -->|"builds retriever"| node_bm25
node_pipeline -->|"builds or loads"| node_vector
node_bm25 -->|"sparse candidates"| node_hybrid
node_vector -->|"dense candidates"| node_hybrid
node_pipeline -->|"assembles retrieval"| node_hybrid
node_hybrid -->|"retrieves variants"| node_multi_query
node_pipeline -->|"creates retriever"| node_multi_query
node_pipeline -->|"reranks candidates"| node_reranker
node_multi_query -->|"returns candidates"| node_reranker
node_pipeline -->|"builds context and answer"| node_generator
node_generator -->|"requests completion"| node_groq
node_pipeline -->|"reads index path"| node_settings
node_generator -->|"reads model settings"| node_settings
node_pipeline -->|"returns answer and sources"| node_main
node_batch -->|"writes JSON"| node_results
node_results -->|"supplies evaluation input"| node_evaluator

click node_main "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/main.py"
click node_batch "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/export_rag_outputs.py"
click node_pipeline "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/pipeline.py"
click node_source_docs "https://github.com/aryanprabhat-spec/mit-gpt/tree/main/data/raw"
click node_loader "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/ingestion/loader.py"
click node_chunker "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/processing/chunker.py"
click node_chunk_store "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/processing/chunker.py"
click node_settings "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/config/settings.py"
click node_bm25 "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/bm25.py"
click node_vector "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/vector_store.py"
click node_hybrid "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/hybrid.py"
click node_multi_query "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/multi_query.py"
click node_reranker "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/reranker.py"
click node_generator "https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/generation/generator.py"
click node_results "https://github.com/aryanprabhat-spec/mit-gpt/tree/main/outputs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_batch,node_pipeline,node_results,node_user toneBlue
class node_source_docs,node_loader,node_chunker,node_chunk_store,node_settings toneAmber
class node_bm25,node_vector,node_hybrid,node_multi_query,node_reranker toneMint
class node_generator toneRose
class node_groq,node_evaluator toneIndigo
```

## Components

### User Interfaces

- [Interactive Q&A](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/main.py): `main.py`
- [Batch Export](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/export_rag_outputs.py): `export_rag_outputs.py`
- [RAG Pipeline](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/pipeline.py): `src/pipeline.py`
- [RAG Results JSON](https://github.com/aryanprabhat-spec/mit-gpt/tree/main/outputs): `outputs`

### Document Indexing

- [Course Notes](https://github.com/aryanprabhat-spec/mit-gpt/tree/main/data/raw): `data/raw`
- [PDF/PPTX Loader](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/ingestion/loader.py): `src/ingestion/loader.py`
- [Semantic Chunking](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/processing/chunker.py): `src/processing/chunker.py`
- [Persisted Chunks](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/processing/chunker.py): `src/processing/chunker.py`
- [Configuration](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/config/settings.py): `src/config/settings.py`

### Retrieval

- [BM25 Retriever](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/bm25.py): `src/retrieval/bm25.py`
- [Dense Vector Store](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/vector_store.py): `src/retrieval/vector_store.py`
- [Hybrid Retrieval](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/hybrid.py): `src/retrieval/hybrid.py`
- [Query Expansion](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/multi_query.py): `src/retrieval/multi_query.py`
- [Cross-Encoder Reranker](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/retrieval/reranker.py): `src/retrieval/reranker.py`

### Answer Generation

- [Grounded Generation](https://github.com/aryanprabhat-spec/mit-gpt/blob/main/src/generation/generator.py): `src/generation/generator.py`

### Other components

- **User**
- **Groq LLM**
- **Evaluation Consumer**

## Connections

- User → Interactive Q&A: asks question (evidence: `README.md`)
- Interactive Q&A → RAG Pipeline: builds pipeline (evidence: `main.py`)
- User → Batch Export: starts export (evidence: `README.md`)
- Batch Export → RAG Pipeline: builds pipeline (evidence: `export_rag_outputs.py`)
- Course Notes → PDF/PPTX Loader: loads files (evidence: `src/ingestion/loader.py`)
- RAG Pipeline → PDF/PPTX Loader: loads documents (evidence: `src/pipeline.py`)
- RAG Pipeline → Semantic Chunking: chunks documents (evidence: `src/pipeline.py`)
- Semantic Chunking → Persisted Chunks: saves and loads (evidence: `src/processing/chunker.py`)
- RAG Pipeline → Persisted Chunks: persists or restores (evidence: `src/pipeline.py`)
- RAG Pipeline → BM25 Retriever: builds retriever (evidence: `src/pipeline.py`)
- RAG Pipeline → Dense Vector Store: builds or loads (evidence: `src/pipeline.py`)
- BM25 Retriever → Hybrid Retrieval: sparse candidates (evidence: `README.md`)
- Dense Vector Store → Hybrid Retrieval: dense candidates (evidence: `README.md`)
- RAG Pipeline → Hybrid Retrieval: assembles retrieval (evidence: `src/pipeline.py`)
- Hybrid Retrieval → Query Expansion: retrieves variants (evidence: `README.md`)
- RAG Pipeline → Query Expansion: creates retriever (evidence: `src/pipeline.py`)
- RAG Pipeline → Cross-Encoder Reranker: reranks candidates (evidence: `src/pipeline.py`)
- Query Expansion → Cross-Encoder Reranker: returns candidates (evidence: `src/pipeline.py`)
- RAG Pipeline → Grounded Generation: builds context and answer (evidence: `src/pipeline.py`)
- Grounded Generation → Groq LLM: requests completion (evidence: `src/generation/generator.py`)
- RAG Pipeline → Configuration: reads index path (evidence: `src/pipeline.py`)
- Grounded Generation → Configuration: reads model settings (evidence: `src/generation/generator.py`)
- RAG Pipeline → Interactive Q&A: returns answer and sources (evidence: `main.py`)
- Batch Export → RAG Results JSON: writes JSON (evidence: `export_rag_outputs.py`)
- RAG Results JSON → Evaluation Consumer: supplies evaluation input (evidence: `README.md`)
