How technion-cs-nlp/llmsknow works
Project Type and Purpose
• This is a research codebase (Python 3.11) for probing large language models (LLMs) to study intrinsic representations of hallucinations, reproducing the experiments from the paper “LLMs Know More Than They Show.”
• It is organized as a data‐processing and ML‐evaluation pipeline, not a client-server app.
File Structure Analysis
• Top‐level:
• data/ — raw CSV datasets (AnswerableMath, MNLI, MovieQA, Winobias, Winogrande, Natural Questions, etc.) and placeholders for TriviaQA and HotpotQA.
• src/ — CLI Python scripts that implement each pipeline stage.
• requirements.txt — Python dependencies, including WandB and HuggingFace.
• src/ contents reveal modular stages:
• generate_model_answers.py → calls the LLM to generate answers
• extract_exact_answer.py → post-processes generations to pull out exact answers
• resampling.py & resampling_merge_runs.py → bootstrap resamples of model outputs
• probe_all_layers_and_tokens.py, probe.py, probe_type_of_error.py, probe_choose_answer.py → classification probes on hidden activations
• logprob_detection.py, p_true_detection.py → baseline error detectors
• compute_correctness.py → final metric computation
• probing_utils.py, resamples_utils.py → shared utility functions
Core Components and Interactions
• Data Ingestion Layer
• Sources: local CSV files in data/ and HuggingFace-loaded datasets.
• Output: consolidated input data fed into the pipeline.
• Model Inference Layer
• Component: generate_model_answers.py
• Input: datasets + model name
• Output: raw model-generated answers (logged via WandB and/or saved locally)
• Answer Extraction Layer
• Component: extract_exact_answer.py
• Input: raw generations
• Output: exact answer tokens per example
• Resampling Layer (Sections 5 & 6)
• Component: resampling.py → bootstrap sampling of generated answers
• resampling_merge_runs.py → consolidates parallel runs
• extract_exact_answer.py (with do_resampling flag) → extracts from resamples
• Probing Layer (Sections 2–4)
• probe_all_layers_and_tokens.py → heatmaps over all layers & tokens
• probe.py → train/evaluate linear classifiers on specific layer/token
• probe_type_of_error.py → error‐type classification
• probe_choose_answer.py → answer‐selection experiments
• Baseline Detection Layer
• logprob_detection.py → detect hallucinations via token log-probs
• p_true_detection.py → detect via model’s predicted probability
• Evaluation & Logging
• compute_correctness.py → aggregates final correctness metrics
• WandB service → logs experiments, metrics, and visualizations
Architectural Patterns & Design Principles
• Pipeline Pattern: Each script represents a stage; outputs feed as inputs to the next.
• Modularity: Utility scripts factor out shared code (probing_utils, resamples_utils).
• CLI-driven: Each step executed via a script with flags (–model, –dataset, etc.).
• External Service Integration: WandB for experiment tracking; HuggingFace Hub for model/data.