# bilalerrabia/rag architecture

> Architecture diagram of the GitHub repository bilalerrabia/rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bilalerrabia/rag
- Repository: https://github.com/bilalerrabia/rag

## Overview

RAG against the Machine indexes a vLLM repository, retrieves relevant source spans for a question, and uses those spans to generate a grounded answer. Its CLI also supports batch search/answer workflows and Recall@k evaluation against supplied datasets. The graph separates ingestion, retrieval, generation, and evaluation, with typed data models as a shared internal layer. The README documents the dual-index design; edges are limited to relationships shown in sampled source or explicitly described there.

Main entry points: [`src/__main__.py`](https://github.com/bilalerrabia/rag/blob/main/src/__main__.py) (RAG CLI), [`src/indexer.py`](https://github.com/bilalerrabia/rag/blob/main/src/indexer.py) (Dual Indexer), [`src/retriever.py`](https://github.com/bilalerrabia/rag/blob/main/src/retriever.py) (Hybrid Search), [`src/evaluator.py`](https://github.com/bilalerrabia/rag/blob/main/src/evaluator.py) (Recall Evaluator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_chunker["File Chunker<br/>[chunker.py]"]
  node_indexer["Dual Indexer<br/>[indexer.py]"]
  node_chunkdata["Chunk Models<br/>[models.py]"]
  node_chunks["Chunk Metadata"]
  node_bm25store[("BM25 Index")]
  node_chromastore[("ChromaDB Collection")]
  node_embedder["Embedding Model"]
end

subgraph group_retrieval["Retrieval"]
  node_hybrid["Hybrid Search<br/>[retriever.py]"]
  node_sourcefiles["Source Files"]
  node_source["Source Models<br/>[models.py]"]
  node_context["Context Loader<br/>[__main__.py]"]
end

subgraph group_generation["Generation"]
  node_answerer["Answer Builder<br/>[answerer.py]"]
  node_llm["Qwen Generator<br/>[llm_model.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_results["Search Results<br/>[models.py]"]
  node_datasets["Question Datasets"]
  node_evaluator["Recall Evaluator<br/>[evaluator.py]"]
  node_answerdata["Answered Datasets"]
end

node_user(("User"))
node_corpus["vLLM Repository"]
node_cli["RAG CLI<br/>[__main__.py]"]

node_user -->|"asks or runs"| node_cli
node_corpus -->|"provides files"| node_indexer
node_cli -->|"invokes indexing"| node_indexer
node_indexer -->|"loads chunks"| node_chunker
node_chunker -->|"creates records"| node_chunkdata
node_indexer -->|"uses records"| node_chunkdata
node_indexer -->|"writes vectors"| node_chromastore
node_indexer -->|"encodes chunks"| node_embedder
node_indexer -->|"writes index"| node_bm25store
node_indexer -->|"writes metadata"| node_chunks
node_cli -->|"searches query"| node_hybrid
node_hybrid -->|"retrieves terms"| node_bm25store
node_hybrid -->|"loads chunk metadata"| node_chunks
node_hybrid -->|"queries vectors"| node_chromastore
node_hybrid -->|"encodes query"| node_embedder
node_hybrid -->|"returns spans"| node_source
node_cli -->|"loads answer context"| node_context
node_context -->|"reads spans"| node_sourcefiles
node_cli -->|"requests answer"| node_answerer
node_answerer -->|"generates response"| node_llm
node_cli -->|"loads questions"| node_datasets
node_cli -->|"records search output"| node_results
node_cli -->|"runs metrics"| node_evaluator
node_evaluator -->|"reads retrievals"| node_results
node_evaluator -->|"compares sources"| node_answerdata

click node_cli "https://github.com/bilalerrabia/rag/blob/main/src/__main__.py"
click node_chunker "https://github.com/bilalerrabia/rag/blob/main/src/chunker.py"
click node_indexer "https://github.com/bilalerrabia/rag/blob/main/src/indexer.py"
click node_chunkdata "https://github.com/bilalerrabia/rag/blob/main/src/models.py"
click node_hybrid "https://github.com/bilalerrabia/rag/blob/main/src/retriever.py"
click node_source "https://github.com/bilalerrabia/rag/blob/main/src/models.py"
click node_context "https://github.com/bilalerrabia/rag/blob/main/src/__main__.py"
click node_answerer "https://github.com/bilalerrabia/rag/blob/main/src/answerer.py"
click node_llm "https://github.com/bilalerrabia/rag/blob/main/src/llm_model.py"
click node_results "https://github.com/bilalerrabia/rag/blob/main/src/models.py"
click node_datasets "https://github.com/bilalerrabia/rag/blob/main/data/datasets/UnansweredQuestions/dataset_docs_public.json"
click node_evaluator "https://github.com/bilalerrabia/rag/blob/main/src/evaluator.py"
click node_answerdata "https://github.com/bilalerrabia/rag/blob/main/data/datasets/AnsweredQuestions/dataset_docs_public.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chunker,node_indexer,node_chunkdata,node_chunks,node_bm25store,node_chromastore,node_embedder,node_user toneBlue
class node_hybrid,node_sourcefiles,node_source,node_context toneAmber
class node_answerer,node_llm toneMint
class node_results,node_datasets,node_evaluator,node_answerdata toneRose
class node_corpus toneIndigo
class node_cli toneTeal
```

## Components

### Ingestion

- [File Chunker](https://github.com/bilalerrabia/rag/blob/main/src/chunker.py): `src/chunker.py`
- [Dual Indexer](https://github.com/bilalerrabia/rag/blob/main/src/indexer.py): `src/indexer.py`
- [Chunk Models](https://github.com/bilalerrabia/rag/blob/main/src/models.py): `src/models.py`
- **Chunk Metadata**
- **BM25 Index**
- **ChromaDB Collection**
- **Embedding Model**

### Retrieval

- [Hybrid Search](https://github.com/bilalerrabia/rag/blob/main/src/retriever.py): `src/retriever.py`
- **Source Files**
- [Source Models](https://github.com/bilalerrabia/rag/blob/main/src/models.py): `src/models.py`
- [Context Loader](https://github.com/bilalerrabia/rag/blob/main/src/__main__.py): `src/__main__.py`

### Generation

- [Answer Builder](https://github.com/bilalerrabia/rag/blob/main/src/answerer.py): `src/answerer.py`
- [Qwen Generator](https://github.com/bilalerrabia/rag/blob/main/src/llm_model.py): `src/llm_model.py`

### Evaluation

- [Search Results](https://github.com/bilalerrabia/rag/blob/main/src/models.py): `src/models.py`
- [Question Datasets](https://github.com/bilalerrabia/rag/blob/main/data/datasets/UnansweredQuestions/dataset_docs_public.json): `data/datasets/UnansweredQuestions/dataset_docs_public.json`
- [Recall Evaluator](https://github.com/bilalerrabia/rag/blob/main/src/evaluator.py): `src/evaluator.py`
- [Answered Datasets](https://github.com/bilalerrabia/rag/blob/main/data/datasets/AnsweredQuestions/dataset_docs_public.json): `data/datasets/AnsweredQuestions/dataset_docs_public.json`

### Other components

- **User**
- **vLLM Repository**
- [RAG CLI](https://github.com/bilalerrabia/rag/blob/main/src/__main__.py): `src/__main__.py`

## Connections

- User → RAG CLI: asks or runs (evidence: `src/__main__.py`)
- vLLM Repository → Dual Indexer: provides files (evidence: `README.md`)
- RAG CLI → Dual Indexer: invokes indexing (evidence: `src/__main__.py`)
- Dual Indexer → File Chunker: loads chunks (evidence: `src/indexer.py`)
- File Chunker → Chunk Models: creates records (evidence: `src/chunker.py`)
- Dual Indexer → Chunk Models: uses records (evidence: `src/indexer.py`)
- Dual Indexer → ChromaDB Collection: writes vectors (evidence: `src/indexer.py`)
- Dual Indexer → Embedding Model: encodes chunks (evidence: `src/indexer.py`)
- Dual Indexer → BM25 Index: writes index (evidence: `src/indexer.py`)
- Dual Indexer → Chunk Metadata: writes metadata (evidence: `src/indexer.py`)
- RAG CLI → Hybrid Search: searches query (evidence: `src/__main__.py`)
- Hybrid Search → BM25 Index: retrieves terms (evidence: `src/retriever.py`)
- Hybrid Search → Chunk Metadata: loads chunk metadata (evidence: `src/retriever.py`)
- Hybrid Search → ChromaDB Collection: queries vectors (evidence: `src/retriever.py`)
- Hybrid Search → Embedding Model: encodes query (evidence: `src/retriever.py`)
- Hybrid Search → Source Models: returns spans (evidence: `src/retriever.py`)
- RAG CLI → Context Loader: loads answer context (evidence: `src/__main__.py`)
- Context Loader → Source Files: reads spans (evidence: `src/__main__.py`)
- RAG CLI → Answer Builder: requests answer (evidence: `src/__main__.py`)
- Answer Builder → Qwen Generator: generates response (evidence: `src/answerer.py`)
- RAG CLI → Question Datasets: loads questions (evidence: `src/__main__.py`)
- RAG CLI → Search Results: records search output (evidence: `src/__main__.py`)
- RAG CLI → Recall Evaluator: runs metrics (evidence: `src/__main__.py`)
- Recall Evaluator → Search Results: reads retrievals (evidence: `src/evaluator.py`)
- Recall Evaluator → Answered Datasets: compares sources (evidence: `src/evaluator.py`)
