How jlab-nlp/retrieval-augmented-correction works
Project type and purpose
• This is an open-source research pipeline (“Retrieval-Augmented-Correction”) rather than a full-stack web app.
• It implements the RAC method from the paper “Efficient LLM Factuality Correction with Retrieval Augmentation.”
• Its primary use case is to ingest benchmark datasets (TruthfulQA, FactScore), perform retrieval-augmented generation and correction of statements, and then evaluate factuality.
File structure
• Top-level directories and files:
• data/ – raw and processed datasets and retrieval outputs
• scripts/ – shell scripts orchestrating various runs (generation, retrieval, RAG, evaluation)
• src/ – core Python modules (retriever, generator, pipeline, evaluators, utils)
• setenv.sh, setup.sh – environment and dependency setup
• README.md, LICENSE, .gitignore
• Key src modules:
• retriever.py, google_search.py – fetches external documents
• generator.py, openai_generator.py – calls LLMs for generation
• pipeline*.py, refactor-pipe*.py – glue logic orchestrating the end-to-end flow
• factscore_*.py, truthfulqa_evaluator.py – evaluation logic
• components.py, task_utils.py, tpu_variables.py, cost_estimator.py – helper functions/config
• The sheer number of scripts indicates the pipeline is driven by shell wrappers around the Python code.
README for additional insights
• The README outlines:
• Environment setup (setenv.sh, setup.sh)
• CLI interface of refactor-pipe.py with flags for enabling/disabling each stage (generation, breakdown, verification, correction, RAG)
• Evaluation workflows for each benchmark