How rahulanand1103/rag-citation works
Project Type and Purpose
• RAG Citation is an open-source Python library (not a web app) that augments Retrieval-Augmented Generation (RAG) pipelines by generating automatic citations via NLP and embedding-based semantic search.
• It exposes two key entry-points: the CiteItem data model (input container) and the Inference class (pipeline orchestrator).
File Structure Analysis
• Top level:
• rag_citation/ – main package
• test/ – unit tests
• Inside rag_citation/:
• base_model/ – abstract and concrete embedding & spaCy NER models
• focus_word/ – logic to extract and schema-validate NER focus words
• score/ – scoring module to rank citation candidates & detect hallucination
• schema.py & cite_item.py – Pydantic-style data models
• inference.py – pipeline controller
• Configuration files: requirements.txt, setup.py, makefile
Key Insights from README
• Pipeline stages: format input → load spaCy & embedding model → NER extraction → semantic search → pair generation → scoring & hallucination check → output
Main Components and Their Interactions
a. Input Layer
• User / client code constructs a CiteItem(answer, context[]).
b. Inference Orchestrator (inference.py)
• Initializes spaCy model & embedding model (via factory methods in base_model/).
• Drives sequential sub-components.
c. Preprocessing & NER (focus_word/)
• Uses spaCy model to extract named entities from the answer.
d. Embedding & Semantic Search (base_model/embedding_model.py)
• Encodes context documents and answer sentences to vector space.
• Computes similarity scores against context.
e. Pair Generation (pair/)
• Matches answer sentences to top-k context chunks.
• Constructs intermediate pair objects with metadata.
f. Scoring & Hallucination Detection (score/score.py)
• Ranks candidate citations, applies threshold.
• Compares entities in answer vs. those found in context to flag hallucination / missing entities.
g. Output Layer
• Returns a structured result:
– citation: list of answer-sentence → cite_document mappings
– hallucination: boolean
– missing: list of un-matched entities
Architectural Patterns and Design Principles
• Modular pipeline: each stage in its own package folder
• Strategy/factory pattern for model selection (embeddings & spaCy)