How naisofly/hallushield works
Project Overview and Purpose
• The project is a proof-of-concept system aimed at detecting hallucinations in large language models in healthcare by classifying responses as either factual or hallucinated.
• Its core value stems from combining sparse autoencoders (via Ember) with tree-based classification models (like XGBoost) to process neural activation features from models such as Meta Llama 3.
• It covers an end-to-end pipeline—from dataset preparation and feature extraction to model training and interpretation.
Analyzing File Structure
• Top-level files include:
• README.md: Contains an extensive description of the project’s purpose, methodology, and pipeline steps.
• meddata.ipynb and visulizations.ipynb: Likely contain exploratory data analysis, dataset preparation steps, and visualization of data/model outputs.
• requirements.txt: Lists dependencies, which can be useful to identify frameworks and libraries.
• The .gitignore ensures repository hygiene but isn’t central to the architecture.
Key Components
a. Data Preparation Module:
• Input: Med-Hallu dataset splitting into non-hallucinated and hallucinated segments.
• Represents preprocessing and data labelling steps.
b. Feature Extraction Module:
• Utilizes the Contrast API (Ember) to extract distinguishing features from LLM-generated responses.
• Indicates the use of sparse autoencoders.
c. Feature Ranking Module:
• Involves selecting top K features and ranking them using GoodFire ReRank along with specific hallucination and non-hallucination queries.
d. Activation Strength Computation:
• Uses the Inspect API to compute the activation strengths for the ranked features.
e. Labeled Dataset Construction:
• Combines activation strengths with their corresponding hallucination labels, setting up the supervised learning dataset.
f. Model Training Module:
• Training of classifiers (Decision Tree, Random Forest, XGBoost), with the selection based on performance metrics (e.g., F1 score).
g. Visualization & Evaluation (represented in notebooks):
• meddata.ipynb and visulizations.ipynb are likely used for initial data exploration, model evaluation, and results visualization.
Relationships and Data Flow
• Flow to the Feature Extraction component where raw input is transformed into neural activation features.
• Features pass to Feature Ranking (leveraging GoodFire ReRank) to filter the most significant features.
• Once features are ranked, pass them to the Inspect API for computing activation strengths.
• The resulting data, including activation strengths and their corresponding labels, feed into the Labeled Dataset Construction.
• This dataset is then used in the Model Training phase, where multiple classifiers are trained and evaluated.
• Feedback from visualizations and evaluation (in the notebooks) might be incorporated to any of the preceding stages for iterative improvements.
Architectural Patterns and Design Principles
• The system follows a modular, pipeline-based architecture with each phase clearly separated:
– Data Ingestion/Preparation
– Feature Engineering
– Model Training and Evaluation
• Emphasis is placed on interpretability by explicitly showing which neural activations correlate with hallucinations.
• The system integrates several external APIs (Contrast API, Goodfire ReRank, and Inspect API) as plug-in components that enhance its functionality.
• The design can be considered an example of a data processing pipeline typical in machine learning projects, specifically oriented toward post-processing LLM outputs.