How minhvuong2000/llmreasoncert works
Project type and purpose
• This is a research-code “pipeline” project for evaluating chain-of-thought (CoT) reasoning of large language models (LLMs) on multi-hop question answering over knowledge graphs.
• It implements two evaluation modules—“discriminative” (judging if a reasoning path is correct) and “generative” (assessing the faithfulness of LLM-generated CoTs via retrieval over KGs)—plus support scripts for preprocessing, vector DB creation, LLM prompting, and result analysis.
File structure
• Top-level scripts & entry points:
• extract_triplet/ → triplet extraction for VectorDB
• discriminative_cert/ → discriminative evaluation code + scripts
• generative_cert/ → generative evaluation code + scripts
• llms/ → LLM wrappers (OpenAI, HF) and adapter classes
• evaluate_results.py, finegrained_analysis.py → final analysis and metrics
• virtuoso_db/ → KG querying layer (Virtuoso)
• scripts/ → SLURM / bash launchers for batch runs
• Config/build artifacts: .env.example, requirements.txt, .gitignore
Extract key insights from the README
• Datasets: CWQ & GrailQA.
• Two modes:
• Discriminative—scores ground-truth vs. negative paths.
• Generative—two-stage LLM prompts + retrieval + fine-grained edit-distance evaluation.