How af011/technical-manual-rag-comparison works
Project Type and Purpose
• This is a research‐oriented Jupyter Notebook project comparing two Retrieval-Augmented Generation (RAG) techniques—reranking and fusion retrieval—applied to the Fervi T999 technical manual.
• The “rag_techniques_comparison.ipynb” drives the entire workflow: loading docs, building indices, running retrieval pipelines, evaluating answer quality.
Core Components
a. Data Ingestion & Preprocessing
• Raw technical manual (PDF or text)
• Text split/parsing module
• Tokenization & normalization
b. Indexing Layer
• BM25 index (e.g., Whoosh, Elasticsearch)
• Vector index (e.g., FAISS, Annoy)
c. Retrieval Pipelines
• Baseline RAG (simple vector search + generative model)
• Reranking pipeline
– First pass: initial retrieval (BM25 or vector)
– Cross-encoder model for reordering top-K docs
• Fusion retrieval pipeline
– Parallel BM25 + vector retrieval
– Score fusion (weighted by α parameter)
d. Language Model / Generator
• LLM interface (e.g., OpenAI, local transformer)
• Prompt templating & response formatting
e. Evaluation & Analysis
• Answer quality metrics (precision, recall, relevance)
• Parameter sweep for α values
• Comparative charts & qualitative summaries
f. Orchestration & Notebook Interface
• Jupyter notebook as control plane
• Sequence of cells driving each pipeline and logging results
Relationships & Data Flows
• Two parallel flows from Query Input:
• Flow A: Query → BM25 index → Top-K docs → Cross-encoder → Reranked docs → LLM → Reranking answers
• Flow B: Query → BM25 & Vector indices in parallel → Score fusion (α) → Top-K fused docs → LLM → Fusion answers
• Evaluation block consumes outputs from Baseline, Reranking, and Fusion pipelines and produces comparative plots and tables
Architectural Patterns & Design Principles
• Modular retrieval components, enabling plug-and-play of different indexers or reranker models
• Separation of concerns: indexing, retrieval, reranking/fusion, generation, and evaluation are distinct layers
• Parameterization: α value controlling fusion behavior, top-K sizing for reranker
• Baseline (→ Vector Index → LLM → Baseline Answers)
• Reranking (→ BM25 or Vector → Cross-Encoder → LLM → Reranking Answers)
• Fusion (→ BM25 & Vector → Fusion Scorer [α] → LLM → Fusion Answers)
5. Fusion component with “α parameter sweep” and the Reranking component with “top-K reordering.”