cdzchen/rag-c09

How cdzchen/rag-c09 works

This repository is a research pipeline for detecting and evaluating hallucinations in retrieval-augmented question answering (RAG). Its intended workflow loads and normalizes QA datasets, constructs normal and three challenge subsets (retrieval failure, evidence conflict, outdated answers), runs retrieval and answer generation, extracts evidence-consistency features, trains a hallucination detector, and evaluates results by challenge type and statistical tests. Shared schema, IO, seeding, and artifact validation support reproducible handoffs. The README says most modules are skeletons; several internal connections below reflect documented capabilities and run order, not inspected implementations.

Main entry points: pipeline.py RAG pipeline · build_features.py Feature builder · cli.py Evaluation workflow · reader.py Dataset reader

Components 27

Data preparation

Dataset reader src/datasets/reader.py
Challenge builder src/datasets/challenge_builder.py
Dataset split src/datasets/split.py

RAG pipeline

Passage index src/retrieval/index.py
BM25 retrieval src/retrieval/bm25.py
Dense retrieval src/retrieval/dense.py
RAG pipeline src/generation/pipeline.py
Generation model src/generation/model.py
Answer decoding src/generation/decode.py
Confidence baseline src/generation/baseline_confidence.py
Evidence demo src/demo/app.py

Feature extraction

Feature builder src/features/build_features.py
Entailment features src/features/entailment.py
Overlap features src/features/overlap.py
Consistency features src/features/consistency.py

Detection

Detector training src/detection/train.py
Hallucination prediction src/detection/predict.py
Probability calibration src/detection/calibrate.py

Evaluation

Evaluation workflow src/evaluation/cli.py
Metrics src/evaluation/metrics.py
Grouped analysis src/evaluation/grouped.py
Significance tests src/evaluation/significance.py

Other components

Researcher
Hugging Face datasets
Artifact IO src/common/io.py
Data contracts src/common/schema.py
Artifact validation src/common/validate.py

Connections 28

Researcher → Dataset reader: selects dataset
Dataset reader → Hugging Face datasets: loads online
Dataset reader → Challenge builder: normalized samples
Challenge builder → Dataset split: challenge samples
Dataset split → Passage index: corpus samples
Passage index → BM25 retrieval: builds corpus
Passage index → Dense retrieval: builds corpus
RAG pipeline → BM25 retrieval: retrieves passages
RAG pipeline → Dense retrieval: retrieves passages
RAG pipeline → Generation model: requests generation
RAG pipeline → Answer decoding: decodes answer
Confidence baseline → Generation model: scores confidence
Researcher → Evidence demo: requests answer
Evidence demo → RAG pipeline: reuses pipeline
RAG pipeline → Feature builder: prediction records
Feature builder → Entailment features: computes features
Feature builder → Overlap features: computes features
Feature builder → Consistency features: computes features
Feature builder → Detector training: feature records
Detector training → Hallucination prediction: trained detector
Detector training → Probability calibration: calibrates scores
Hallucination prediction → Evaluation workflow: prediction records
Evaluation workflow → Metrics: computes metrics
Evaluation workflow → Grouped analysis: groups results
Evaluation workflow → Significance tests: tests differences
Artifact IO → Artifact validation: reads artifacts
Artifact IO → Data contracts: uses metadata key
Artifact validation → Data contracts: checks fields

More Python diagrams: naxiaoduo/scammer.skill · nicolik/segm_ident_vertebrae_cnn_kmeans_knn · nlrwindsystems/weis · opentire/opentirepython · power-agent/powerskills

Drawn by GitDiagram on September 29, 2026 from the repository's file tree, README and source files. Read as Markdown ·