How yuxiaw/factcheck-gpt works
Project Type and Purpose
• This is a Python-based, end-to-end document-level fact-checking pipeline built around generative LLMs.
• It includes:
• A core processing pipeline for LLM-driven subtasks (decomposition, checkworthiness, retrieval, stance, verification & editing).
• An annotation tool (static HTML + example JSON inputs/outputs) for human labeling.
• Datasets (Factcheck-GPT JSONL, FactBench, subtasks), baseline implementations, and evaluation notebooks.
High-Level File-Structure Analysis
• Top-level:
• src/ – core pipeline modules and utilities
• annotation_tool/ – front-end UI for annotation (HTML + examples)
• subtasks/ – JSONL definitions for each subtask dataset
• Factbench.jsonl & factcheck-GPT-benchmark.jsonl – additional evaluation datasets
• reproduce_tutorial.ipynb – end-to-end demo and evaluation
• Core orchestration: src/pipeline.py calls individual subtask modules in sequence.
• Utilities: src/utils contains data handling, prompts, OpenAI API wrappers, evaluation helpers.
Key README Insights
• The pipeline breaks down fact-checking into six stages: decomposition, decontextualization, check-worthiness, retrieval, stance detection, and automated correction.
• Uses OpenAI’s GPT (via src/utils/openaiAPI.py and src/run_langchain.py) and standard ML/NLP libraries.
• Provides an annotation tool repository for manual dataset creation.
Components
a. Data Sources & Artifacts (blue)
• Input document(s) to check
• Subtasks JSONLs (in subtasks/)
• Factcheck-GPT and FactBench datasets
b. Core Pipeline (green)
• Orchestrator: pipeline.py → check_document(), check_documents()
• decompose.py
• (de)contextualization logic (inside decompose or utils)
• checkworthy.py
• retrieve.py
• identify_stance.py
• verify.py (fact verification + revision)
• Baselines: baselines.py for reference implementations
c. Shared Utilities (light green)
• data_util.py (data I/O)
• prompt.py (prompt templates)
• openaiAPI.py (LLM calls)
• eval_utils.py, nli.py (evaluation & inference helpers)
d. External Services (orange)
• OpenAI API / GPT-3.5-turbo / Langchain chains (via run_langchain.py)
• Potential external knowledge sources or search APIs in retrieve.py
e. Annotation Tool (purple)
• Static HTML screens (screen1.html, screen2.html, screen3.html)
• Example JSON inputs/outputs for human labelers
• Python helper generate_screen3_input.py
f. Evaluation & Demo (grey)
• reproduce_tutorial.ipynb for running end-to-end checks and visualizing results
• fig/ pipeline.pdf and result charts
Relationships and Data Flows
• Pipeline outputs labeled claims and logs → fed into Evaluation notebooks and fig/ charts.
• Annotation Tool reads subtasks JSONL → human annotators fill forms → writes back JSONL → used to augment datasets.
Architectural Patterns & Principles
• Modular pipeline: each subtask is a self-contained component.
• Single Responsibility: utilities abstract common functions (I/O, prompts, API calls).
• Plugin-style extensibility: new subtask modules or model backends can be dropped in.
• Separation of concerns: annotation front-end decoupled from core processing.