How oneal2000/mind works
Project Type and Purpose
• This is a research‐oriented machine-learning pipeline (“MIND”) for unsupervised hallucination detection in transformer‐based LLMs, coupled with the HELM benchmark for evaluation.
• The repository is organized around a data‐processing/training/evaluation workflow rather than a classic client–server or UI-driven application.
File Structure Analysis
• Top‐level scripts:
• generate_data.py → auto-annotates Wikipedia sentences with hallucinations
• generate_hd.py → extracts hidden-state features for model training
• detection_score.py → evaluates trained classifier on the HELM benchmark
• auto-labeled/
• train/ → dataset.py (data loader) + train.py (classifier training)
• wiki/ → raw Wikipedia splits (train/valid/test)
• helm/ → HELM benchmark data: model outputs, prompts, hidden states, activations
• utils/ → shared code (gen.py, model.py) for invoking LLMs, feature routines
• requirements.txt → Python libraries (torch, transformers, scikit-learn, spacy, etc.)
README Insights
• Four-step pipeline:
1. generate_data.py
2. generate_hd.py
3. auto-labeled/train/train.py
4. detection_score.py
• External dependencies: any Transformer-based LLM (via HuggingFace Transformers) plus Spacy for entity extraction.
• HELM benchmark data organized per model, with prompt→output→hidden states→activation values.
1. Wikipedia Data Loader (utils + auto-labeled/wiki)
2. Auto-Annotation Module (generate_data.py + Spacy + Transformers)
3. Feature Extraction Module (generate_hd.py + Transformers hooks + utils)
4. Classifier Training Module (auto-labeled/train/dataset.py & train.py + scikit-learn or PyTorch MLP)
5. HELM Benchmark Data Store (helm/data, helm/hd, helm/hd_act)
6. Evaluation Module (detection_score.py + output Excel reports)
7. Shared Utilities Layer (utils/gen.py, utils/model.py)
• Wikipedia Data Loader → Auto-Annotation Module
• Auto-Annotation Module → auto-labeled output JSON
• auto-labeled JSON → Feature Extraction Module → hidden-state JSON
• hidden-state JSON → Classifier Training Module → trained model artifact
• HELM Benchmark Data Store → Evaluation Module + trained model → performance reports
• Shared Utilities Layer ← all scripts (generate_data.py, generate_hd.py, train.py, detection_score.py)
• External LLMs ↔ Auto-Annotation Module & Feature Extraction Module
c. Architectural Patterns and Principles:
• Pipeline pattern: each step produces artifacts consumed by the next
• Modularization: clear separation between data annotation, feature extraction, training, evaluation
• Reusability: utils/ contains common code for invoking models and processing hidden states
• Configuration by script parameters and JSON outputs for reproducibility
d. Key Technologies & Frameworks:
• Python 3.9
• Spacy (entity extraction)
• HuggingFace Transformers (LLM inference + hidden states)
• PyTorch (ML model, MLP classifier)
• scikit-learn (metrics, train/test splits)
• pandas/numpy (data handling)
• Excel reports (result_*.xlsx)