# yuxiaw/factcheck-gpt architecture

> Architecture diagram of the GitHub repository yuxiaw/factcheck-gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuxiaw/factcheck-gpt
- Repository: https://github.com/yuxiaw/factcheck-gpt

## Overview

##### Project Type and Purpose
- This is a Python-based, end-to-end document-level fact-checking pipeline built around generative LLMs.
- It includes:
- A core processing pipeline for LLM-driven subtasks (decomposition, checkworthiness, retrieval, stance, verification & editing).
- An annotation tool (static HTML + example JSON inputs/outputs) for human labeling.
- Datasets (Factcheck-GPT JSONL, FactBench, subtasks), baseline implementations, and evaluation notebooks.

##### High-Level File-Structure Analysis
- Top-level:
- src/ – core pipeline modules and utilities
- annotation_tool/ – front-end UI for annotation (HTML + examples)
- subtasks/ – JSONL definitions for each subtask dataset
- Factbench.jsonl & factcheck-GPT-benchmark.jsonl – additional evaluation datasets
- reproduce_tutorial.ipynb – end-to-end demo and evaluation
- Core orchestration: src/pipeline.py calls individual subtask modules in sequence.
- Utilities: src/utils contains data handling, prompts, OpenAI API wrappers, evaluation helpers.

##### Key README Insights
- The pipeline breaks down fact-checking into six stages: decomposition, decontextualization, check-worthiness, retrieval, stance detection, and automated correction.
- Uses OpenAI’s GPT (via src/utils/openaiAPI.py and src/run_langchain.py) and standard ML/NLP libraries.
- Provides an annotation tool repository for manual dataset creation.

##### Components
a. Data Sources & Artifacts (blue)
- Input document(s) to check
- Subtasks JSONLs (in subtasks/)
- Factcheck-GPT and FactBench datasets
b. Core Pipeline (green)
- Orchestrator: pipeline.py → check_document(), check_documents()
- decompose.py
- (de)contextualization logic (inside decompose or utils)
- checkworthy.py
- retrieve.py
- identify_stance.py
- verify.py (fact verification + revision)
- Baselines: baselines.py for reference implementations
c. Shared Utilities (light green)
- data_util.py (data I/O)
- prompt.py (prompt templates)
- openaiAPI.py (LLM calls)
- eval_utils.py, nli.py (evaluation & inference helpers)
d. External Services (orange)
- OpenAI API / GPT-3.5-turbo / Langchain chains (via run_langchain.py)
- Potential external knowledge sources or search APIs in retrieve.py
e. Annotation Tool (purple)
- Static HTML screens (screen1.html, screen2.html, screen3.html)
- Example JSON inputs/outputs for human labelers
- Python helper generate_screen3_input.py
f. Evaluation & Demo (grey)
- reproduce_tutorial.ipynb for running end-to-end checks and visualizing results
- fig/ pipeline.pdf and result charts

##### Relationships and Data Flows
- Pipeline outputs labeled claims and logs → fed into Evaluation notebooks and fig/ charts.
- Annotation Tool reads subtasks JSONL → human annotators fill forms → writes back JSONL → used to augment datasets.

##### Architectural Patterns & Principles
- Modular pipeline: each subtask is a self-contained component.
- Single Responsibility: utilities abstract common functions (I/O, prompts, API calls).
- Plugin-style extensibility: new subtask modules or model backends can be dropped in.
- Separation of concerns: annotation front-end decoupled from core processing.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources & Artifacts
    subgraph "Data Sources & Artifacts"
        style DataSources fill:#EDF5FF,stroke:#0366D6,stroke-width:2px
        DS1["subtasks/ (JSONL)"]:::data
        DS2["Factbench.jsonl"]:::data
        DS3["factcheck-GPT-benchmark.jsonl"]:::data
    end

    %% Core Pipeline
    subgraph "Core Pipeline" 
        style CorePipeline fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px
        ORC["pipeline.py"]:::processor
        DEC["decompose.py"]:::processor
        CW["checkworthy.py"]:::processor
        RET["retrieve.py"]:::processor
        STA["identify_stance.py"]:::processor
        VER["verify.py"]:::processor
        BAS["baselines.py"]:::processor
    end

    %% Shared Utilities
    subgraph "Shared Utilities"
        style Utils fill:#F1F8E9,stroke:#558B2F,stroke-width:2px
        DU["data_util.py"]:::util
        PR["prompt.py"]:::util
        APIW["openaiAPI.py"]:::util
        EU["eval_utils.py"]:::util
        NLI["nli.py"]:::util
    end

    %% External Services
    subgraph "External Services"
        style Ext fill:#FFF3E0,stroke:#EF6C00,stroke-width:2px
        LC["run_langchain.py"]:::external
        OPENAI["OpenAI API/GPT-3.5"]:::external
    end

    %% Annotation Tool
    subgraph "Annotation Tool"
        style Annot fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px
        A1["screen1.html"]:::annotation
        A2["screen2.html"]:::annotation
        A3["screen3.html"]:::annotation
        AE["example/0_screen1.json…"]:::annotation
        AG["generate_screen3_input.py"]:::annotation
    end

    %% Evaluation & Demo
    subgraph "Evaluation & Demo"
        style Eval fill:#ECEFF1,stroke:#455A64,stroke-width:2px
        NB["reproduce_tutorial.ipynb"]:::eval
        FIG["fig/pipeline.pdf"]:::eval
    end

    %% Relationships
    DS1 -.->|reads| ORC
    DS2 -.->|reads| ORC
    DS3 -.->|reads| ORC

    ORC -->|"calls"| DEC
    DEC -->|"calls"| CW
    CW -->|"calls"| RET
    RET -->|"calls"| STA
    STA -->|"calls"| VER
    ORC -->|"invokes baseline"| BAS

    %% Submodules to Utilities
    DEC -->|uses| DU
    DEC -->|uses| PR
    CW -->|uses| DU
    CW -->|uses| PR
    RET -->|uses| DU
    RET -->|uses| PR
    STA -->|uses| DU
    STA -->|uses| NLI
    VER -->|uses| DU
    VER -->|uses| PR
    BAS -->|uses| DU

    %% Submodules to External
    DEC -->|LLM call| APIW
    DEC -->|chain| LC
    CW -->|LLM call| APIW
    RET -->|search API| OPENAI
    RET -->|chain| LC
    STA -->|LLM call| APIW
    VER -->|LLM call| APIW

    %% Pipeline outputs to Evaluation
    VER -->|"outputs"| NB
    VER -->|"outputs"| FIG
    BAS -->|"outputs"| NB

    %% Annotation flows
    DS1 -.->|template| A1
    A1 -.->|fills| DS1
    DS1 -.->|template| A2
    A2 -.->|fills| DS1
    A3 -.->|fills| DS1
    AG -->|produces| A3

    %% Click Events
    click DS1 "https://github.com/yuxiaw/factcheck-gpt/tree/main/subtasks/"
    click ORC "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/pipeline.py"
    click DEC "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/decompose.py"
    click CW "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/checkworthy.py"
    click RET "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/retrieve.py"
    click STA "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/identify_stance.py"
    click VER "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/verify.py"
    click BAS "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/baselines.py"
    click DU "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/utils/data_util.py"
    click PR "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/utils/prompt.py"
    click APIW "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/utils/openaiAPI.py"
    click EU "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/utils/eval_utils.py"
    click NLI "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/utils/nli.py"
    click LC "https://github.com/yuxiaw/factcheck-gpt/blob/main/src/run_langchain.py"
    click A1 "https://github.com/yuxiaw/factcheck-gpt/blob/main/annotation_tool/screen1.html"
    click A2 "https://github.com/yuxiaw/factcheck-gpt/blob/main/annotation_tool/screen2.html"
    click A3 "https://github.com/yuxiaw/factcheck-gpt/blob/main/annotation_tool/screen3.html"
    click AE "https://github.com/yuxiaw/factcheck-gpt/blob/main/annotation_tool/example/0_screen1.json"
    click AG "https://github.com/yuxiaw/factcheck-gpt/blob/main/annotation_tool/example/generate_screen3_input.py"
    click NB "https://github.com/yuxiaw/factcheck-gpt/blob/main/reproduce_tutorial.ipynb"
    click FIG "https://github.com/yuxiaw/factcheck-gpt/blob/main/fig/pipeline.pdf"

    %% Styles
    classDef data fill:#BBDEFB,stroke:#0D47A1,color:#0D47A1
    classDef processor fill:#C8E6C9,stroke:#1B5E20,color:#1B5E20
    classDef util fill:#DCEDC8,stroke:#33691E,color:#33691E
    classDef external fill:#FFE0B2,stroke:#E65100,color:#E65100
    classDef annotation fill:#E1BEE7,stroke:#4A148C,color:#4A148C
    classDef eval fill:#CFD8DC,stroke:#263238,color:#263238
```
