# hsmshubham007-code/week5task1 architecture

> Architecture diagram of the GitHub repository hsmshubham007-code/week5task1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hsmshubham007-code/week5task1
- Repository: https://github.com/hsmshubham007-code/week5task1

## Overview

NovaTech is a policy-document RAG assistant: users ask questions in a Streamlit chat and receive evidence-grounded answers or refusals with source/page citations. Its data path extracts PDF pages, creates differently sized chunks, and builds Chroma collections; the QA runtime retrieves chunks, selects and scores evidence, then answers or refuses. Separate evaluation scripts compare baseline, hybrid, and reranked retrieval and test generation. The README documents capabilities beyond the sampled runtime excerpts; links based on those capabilities cite the README.

Main entry points: [`data/golden_set/golden_set.csv`](https://github.com/hsmshubham007-code/week5task1/blob/master/data/golden_set/golden_set.csv) (Golden questions), [`storage/chroma`](https://github.com/hsmshubham007-code/week5task1/tree/master/storage/chroma) (Vector collections), [`src/generation/grounded_qa.py`](https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py) (Grounded QA runtime), [`app.py`](https://github.com/hsmshubham007-code/week5task1/blob/master/app.py) (Streamlit chat)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Corpus ingestion"]
  node_pdfs["Policy PDFs"]
  node_extract["PDF extraction<br/>[extract_pdfs.py]"]
  node_pages["Extracted pages"]
  node_chunk["Token chunking<br/>[chunk_documents.py]"]
  node_chunks["Chunk datasets"]
  node_index["Chroma builder<br/>[build_chroma.py]"]
end

subgraph group_qa["Grounded QA"]
  node_chroma[("Vector collections")]
  node_qa_runtime["Grounded QA runtime<br/>[grounded_qa.py]"]
  node_retrieve["Top-five retrieval<br/>[grounded_qa.py]"]
  node_evidence["Evidence selection<br/>[grounded_qa.py]"]
  node_answer["Answer and citation<br/>[grounded_qa.py]"]
end

subgraph group_eval["Evaluation"]
  node_gold["Golden questions<br/>[golden_set.csv]"]
  node_baseline["Recall evaluation<br/>[evaluate_recall.py]"]
  node_hybrid["BM25 hybrid evaluation<br/>[evaluate_hybrid.py]"]
  node_rerank["Reranker evaluation"]
  node_generation_eval["Generation evaluation"]
  node_full_eval["Full evaluation<br/>[evaluate_all.py]"]
end

node_user(("Policy questioner"))
node_ui["Streamlit chat<br/>[app.py]"]

node_user -->|"asks"| node_ui
node_ui -->|"calls"| node_qa_runtime
node_pdfs -->|"provides PDFs"| node_extract
node_extract -->|"writes pages"| node_pages
node_pages -->|"reads pages"| node_chunk
node_chunk -->|"writes chunks"| node_chunks
node_chunks -->|"reads chunks"| node_index
node_index -->|"builds collections"| node_chroma
node_qa_runtime -->|"runs"| node_retrieve
node_retrieve -->|"queries"| node_chroma
node_retrieve -->|"supplies chunks"| node_evidence
node_evidence -->|"grounds or refuses"| node_answer
node_answer -->|"returns result"| node_ui
node_gold -->|"evaluated against"| node_baseline
node_gold -->|"evaluated against"| node_hybrid
node_gold -->|"evaluated against"| node_rerank
node_gold -->|"evaluated against"| node_generation_eval
node_gold -->|"evaluated against"| node_full_eval
node_baseline -->|"queries"| node_chroma
node_hybrid -->|"queries vectors"| node_chroma
node_rerank -->|"retrieves candidates"| node_chroma
node_generation_eval -->|"evaluates answers"| node_qa_runtime
node_full_eval -->|"evaluates generation"| node_qa_runtime

click node_ui "https://github.com/hsmshubham007-code/week5task1/blob/master/app.py"
click node_pdfs "https://github.com/hsmshubham007-code/week5task1/tree/master/data/documents"
click node_extract "https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/extract_pdfs.py"
click node_pages "https://github.com/hsmshubham007-code/week5task1/tree/master/data/processed"
click node_chunk "https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/chunk_documents.py"
click node_chunks "https://github.com/hsmshubham007-code/week5task1/tree/master/data/chunks"
click node_index "https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/build_chroma.py"
click node_chroma "https://github.com/hsmshubham007-code/week5task1/tree/master/storage/chroma"
click node_qa_runtime "https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py"
click node_retrieve "https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py"
click node_evidence "https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py"
click node_answer "https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py"
click node_gold "https://github.com/hsmshubham007-code/week5task1/blob/master/data/golden_set/golden_set.csv"
click node_baseline "https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_recall.py"
click node_hybrid "https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_hybrid.py"
click node_rerank "https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_reranker.py"
click node_generation_eval "https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_generation.py"
click node_full_eval "https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_all.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pdfs,node_extract,node_pages,node_chunk,node_chunks,node_index toneBlue
class node_chroma,node_qa_runtime,node_retrieve,node_evidence,node_answer toneAmber
class node_gold,node_baseline,node_hybrid,node_rerank,node_generation_eval,node_full_eval toneMint
class node_user toneIndigo
class node_ui toneTeal
```

## Components

### Corpus ingestion

- [Policy PDFs](https://github.com/hsmshubham007-code/week5task1/tree/master/data/documents): `data/documents`
- [PDF extraction](https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/extract_pdfs.py): `src/ingestion/extract_pdfs.py`
- [Extracted pages](https://github.com/hsmshubham007-code/week5task1/tree/master/data/processed): `data/processed`
- [Token chunking](https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/chunk_documents.py): `src/ingestion/chunk_documents.py`
- [Chunk datasets](https://github.com/hsmshubham007-code/week5task1/tree/master/data/chunks): `data/chunks`
- [Chroma builder](https://github.com/hsmshubham007-code/week5task1/blob/master/src/ingestion/build_chroma.py): `src/ingestion/build_chroma.py`

### Grounded QA

- [Vector collections](https://github.com/hsmshubham007-code/week5task1/tree/master/storage/chroma): `storage/chroma`
- [Grounded QA runtime](https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py): `src/generation/grounded_qa.py`
- [Top-five retrieval](https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py): `src/generation/grounded_qa.py`
- [Evidence selection](https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py): `src/generation/grounded_qa.py`
- [Answer and citation](https://github.com/hsmshubham007-code/week5task1/blob/master/src/generation/grounded_qa.py): `src/generation/grounded_qa.py`

### Evaluation

- [Golden questions](https://github.com/hsmshubham007-code/week5task1/blob/master/data/golden_set/golden_set.csv): `data/golden_set/golden_set.csv`
- [Recall evaluation](https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_recall.py): `evals/evaluate_recall.py`
- [BM25 hybrid evaluation](https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_hybrid.py): `evals/evaluate_hybrid.py`
- [Reranker evaluation](https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_reranker.py): `evals/evaluate_reranker.py`
- [Generation evaluation](https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_generation.py): `evals/evaluate_generation.py`
- [Full evaluation](https://github.com/hsmshubham007-code/week5task1/blob/master/evals/evaluate_all.py): `evals/evaluate_all.py`

### Other components

- **Policy questioner**
- [Streamlit chat](https://github.com/hsmshubham007-code/week5task1/blob/master/app.py): `app.py`

## Connections

- Policy questioner → Streamlit chat: asks (evidence: `README.md`)
- Streamlit chat → Grounded QA runtime: calls (evidence: `app.py`)
- Policy PDFs → PDF extraction: provides PDFs (evidence: `src/ingestion/extract_pdfs.py`)
- PDF extraction → Extracted pages: writes pages (evidence: `src/ingestion/extract_pdfs.py`)
- Extracted pages → Token chunking: reads pages (evidence: `src/ingestion/chunk_documents.py`)
- Token chunking → Chunk datasets: writes chunks (evidence: `src/ingestion/chunk_documents.py`)
- Chunk datasets → Chroma builder: reads chunks (evidence: `src/ingestion/build_chroma.py`)
- Chroma builder → Vector collections: builds collections (evidence: `src/ingestion/build_chroma.py`)
- Grounded QA runtime → Top-five retrieval: runs (evidence: `README.md`)
- Top-five retrieval → Vector collections: queries (evidence: `README.md`)
- Top-five retrieval → Evidence selection: supplies chunks (evidence: `README.md`)
- Evidence selection → Answer and citation: grounds or refuses (evidence: `README.md`)
- Answer and citation → Streamlit chat: returns result (evidence: `app.py`)
- Golden questions → Recall evaluation: evaluated against (evidence: `evals/evaluate_recall.py`)
- Golden questions → BM25 hybrid evaluation: evaluated against (evidence: `evals/evaluate_hybrid.py`)
- Golden questions → Reranker evaluation: evaluated against (evidence: `evals/evaluate_reranker.py`)
- Golden questions → Generation evaluation: evaluated against (evidence: `README.md`)
- Golden questions → Full evaluation: evaluated against (evidence: `evals/evaluate_all.py`)
- Recall evaluation → Vector collections: queries (evidence: `evals/evaluate_recall.py`)
- BM25 hybrid evaluation → Vector collections: queries vectors (evidence: `evals/evaluate_hybrid.py`)
- Reranker evaluation → Vector collections: retrieves candidates (evidence: `evals/evaluate_reranker.py`)
- Generation evaluation → Grounded QA runtime: evaluates answers (evidence: `README.md`)
- Full evaluation → Grounded QA runtime: evaluates generation (evidence: `README.md`)
