# minhvuong2000/llmreasoncert architecture

> Architecture diagram of the GitHub repository minhvuong2000/llmreasoncert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minhvuong2000/llmreasoncert
- Repository: https://github.com/minhvuong2000/llmreasoncert

## Overview

##### Project type and purpose
- This is a research-code “pipeline” project for evaluating chain-of-thought (CoT) reasoning of large language models (LLMs) on multi-hop question answering over knowledge graphs.
- It implements two evaluation modules—“discriminative” (judging if a reasoning path is correct) and “generative” (assessing the faithfulness of LLM-generated CoTs via retrieval over KGs)—plus support scripts for preprocessing, vector DB creation, LLM prompting, and result analysis.

##### File structure
- Top-level scripts & entry points:
- extract_triplet/       → triplet extraction for VectorDB
- discriminative_cert/   → discriminative evaluation code + scripts
- generative_cert/       → generative evaluation code + scripts
- llms/                  → LLM wrappers (OpenAI, HF) and adapter classes
- evaluate_results.py, finegrained_analysis.py → final analysis and metrics
- virtuoso_db/           → KG querying layer (Virtuoso)
- scripts/               → SLURM / bash launchers for batch runs
- Config/build artifacts: .env.example, requirements.txt, .gitignore

##### Extract key insights from the README
- Datasets: CWQ & GrailQA.
- Two modes:
- Discriminative—scores ground-truth vs. negative paths.
- Generative—two-stage LLM prompts + retrieval + fine-grained edit-distance evaluation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        CWQ["CWQ Dataset"]:::data
        GrailQA["GrailQA Dataset"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        subgraph "preprocess_data"
            cwq_graph["cwq_graph.py"]:::preprocessing
            grailqa_graph["grailqa_graph.py"]:::preprocessing
            ground_truth["ground_truth_paths.py"]:::preprocessing
            sparql["sparql.py"]:::preprocessing
            split_gt["splitted_ground_truth_paths.py"]:::preprocessing
            subgraph_disc["subgraph_discriminative_cert.py"]:::preprocessing
        end
        subgraph "extract_subgraph"
            pre_data["preprocess_data.py"]:::preprocessing
            loader["graph_loader.py"]:::preprocessing
            deal_cvt["deal_cvt.py"]:::preprocessing
        end
        triplet["extract_triplet.py"]:::preprocessing
    end

    %% Knowledge Graph Layer
    subgraph "Knowledge Graph Layer"
        Virtuoso["Virtuoso Connector"]:::db
    end

    %% VectorDB Layer
    subgraph "VectorDB Layer"
        FAISS["FAISS Index (Embeddings)"]:::db
    end

    %% LLM Layer
    subgraph "LLM Layer"
        base_lm["base_language_model.py"]:::llm
        hf_model["base_hf_causal_model.py"]:::llm
        chatgpt["chatgpt.py"]:::llm
        conv_prompt["conv_prompt.py"]:::llm
        adapter["model_adapter.py"]:::llm
        gen_entry["llm_generation.py"]:::llm
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        subgraph "Discriminative" 
            disc_prompts["discriminative_prompts.py"]:::eval
            disc_utils["utils.py"]:::eval
            submit_disc["submit_discriminative_cert.sh"]:::eval
            summary_disc["summary_results.py"]:::eval
            disc_entry["discriminative-cert.py"]:::eval
        end
        subgraph "Generative"
            gen_prompts["prompts.py"]:::eval
            gen_utils["utils.py"]:::eval
            data_analytic["data_analytic.py"]:::eval
            eval_llm["eval_llm.sh"]:::eval
            chatgpt_gen["chatgpt_generation.sh"]:::eval
            gen_entry2["generative-cert.py"]:::eval
        end
    end

    %% Analysis Layer
    subgraph "Analysis Layer"
        eval_results["evaluate_results.py"]:::analysis
        fine_grained["finegrained_analysis.py"]:::analysis
        fine_gen["finegrained_gen_cert.py"]:::analysis
    end

    %% Orchestration Scripts
    subgraph "Orchestration Scripts"
        disc_script["submit_discriminative_cert.sh"]:::scripts
        sum_script["summary_results.py"]:::scripts
        gen_subgraph["subgraph.sh"]:::scripts
        gen_prompt["llm_prompting.sh"]:::scripts
        gen_triplet["extract_triplet.sh"]:::scripts
        job_eval1["job_eval_llm.sh"]:::scripts
        job_eval2["job_eval_llm_finegrained.sh"]:::scripts
        run_sh["run.sh"]:::scripts
    end

    %% External Services
    OpenAI["OpenAI API"]:::external
    HFHub["HuggingFace Hub"]:::external

    %% Data Flow
    CWQ --> preprocess_data
    GrailQA --> preprocess_data

    preprocess_data --> Virtuoso
    preprocess_data --> FAISS
    preprocess_data --> disc_entry
    preprocess_data --> gen_entry2

    disc_entry --> chatgpt
    gen_entry2 --> chatgpt

    chatgpt --> OpenAI
    chatgpt --> HFHub
    chatgpt --> FAISS

    disc_entry --> eval_results
    gen_entry2 --> eval_results

    eval_results --> fine_grained
    eval_results --> fine_gen

    disc_script --> disc_entry
    sum_script --> disc_entry
    gen_subgraph --> preprocess_data
    gen_prompt --> chatgpt
    gen_triplet --> triplet
    job_eval1 --> gen_entry2
    job_eval2 --> gen_entry2
    run_sh --> gen_entry2

    %% Click Events
    click cwq_graph "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/cwq_graph.py"
    click grailqa_graph "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/grailqa_graph.py"
    click ground_truth "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/ground_truth_paths.py"
    click sparql "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/sparql.py"
    click split_gt "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/splitted_ground_truth_paths.py"
    click subgraph_disc "https://github.com/minhvuong2000/llmreasoncert/blob/main/preprocess_data/subgraph_discriminative_cert.py"
    click pre_data "https://github.com/minhvuong2000/llmreasoncert/blob/main/extract_subgraph/preprocess_data.py"
    click loader "https://github.com/minhvuong2000/llmreasoncert/blob/main/extract_subgraph/graph_loader.py"
    click deal_cvt "https://github.com/minhvuong2000/llmreasoncert/blob/main/extract_subgraph/deal_cvt.py"
    click triplet "https://github.com/minhvuong2000/llmreasoncert/blob/main/extract_triplet/extract_triplet.py"
    click Virtuoso "https://github.com/minhvuong2000/llmreasoncert/blob/main/virtuoso_db/virtuoso.py"
    click base_lm "https://github.com/minhvuong2000/llmreasoncert/blob/main/llms/base_language_model.py"
    click hf_model "https://github.com/minhvuong2000/llmreasoncert/blob/main/llms/base_hf_causal_model.py"
    click chatgpt "https://github.com/minhvuong2000/llmreasoncert/blob/main/llms/chatgpt.py"
    click conv_prompt "https://github.com/minhvuong2000/llmreasoncert/blob/main/llms/conv_prompt.py"
    click adapter "https://github.com/minhvuong2000/llmreasoncert/blob/main/llms/model_adapter.py"
    click gen_entry "https://github.com/minhvuong2000/llmreasoncert/blob/main/llm_generation.py"
    click disc_prompts "https://github.com/minhvuong2000/llmreasoncert/blob/main/discriminative_cert/discriminative_prompts.py"
    click disc_utils "https://github.com/minhvuong2000/llmreasoncert/blob/main/discriminative_cert/utils/utils.py"
    click submit_disc "https://github.com/minhvuong2000/llmreasoncert/blob/main/discriminative_cert/scripts/submit_discriminative_cert.sh"
    click summary_disc "https://github.com/minhvuong2000/llmreasoncert/blob/main/discriminative_cert/scripts/summary_results.py"
    click disc_entry "https://github.com/minhvuong2000/llmreasoncert/blob/main/discriminative-cert.py"
    click gen_prompts "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative_cert/prompts.py"
    click gen_utils "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative_cert/utils/utils.py"
    click data_analytic "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative_cert/utils/data_analytic.py"
    click eval_llm "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative_cert/scripts/eval_llm.sh"
    click chatgpt_gen "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative_cert/scripts/generation/chatgpt_generation.sh"
    click gen_entry2 "https://github.com/minhvuong2000/llmreasoncert/blob/main/generative-cert.py"
    click eval_results "https://github.com/minhvuong2000/llmreasoncert/blob/main/evaluate_results.py"
    click fine_grained "https://github.com/minhvuong2000/llmreasoncert/blob/main/finegrained_analysis.py"
    click fine_gen "https://github.com/minhvuong2000/llmreasoncert/blob/main/finegrained_gen_cert.py"
    click disc_script "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/disc-cert/submit_discriminative_cert.sh"
    click sum_script "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/disc-cert/summary_results.py"
    click gen_subgraph "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/subgraph.sh"
    click gen_prompt "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/llm_prompting.sh"
    click gen_triplet "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/extract_triplet.sh"
    click job_eval1 "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/job_eval_llm.sh"
    click job_eval2 "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/job_eval_llm_finegrained.sh"
    click run_sh "https://github.com/minhvuong2000/llmreasoncert/blob/main/scripts/gen-cert/run.sh"

    %% Styles
    classDef data fill:#D0E6F7,stroke:#024;stroke-width:1px
    classDef preprocessing fill:#B3D9FF,stroke:#036;stroke-width:1px
    classDef db fill:#C8F7C5,stroke:#070;stroke-width:1px
    classDef llm fill:#D5F5E3,stroke:#088;stroke-width:1px
    classDef eval fill:#FCE8B2,stroke:#E67E22,stroke-width:1px
    classDef analysis fill:#E8DAEF,stroke:#8E44AD,stroke-width:1px
    classDef scripts fill:#ECECEC,stroke:#555,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#999,stroke-width:1px,stroke-dasharray: 5 5
```
