How vectara/hallucination-leaderboard works
Key Components to Include in the System Design:
Data Flow Components
• Input Processing System
• Document Source (CNN/Daily Mail Corpus)
• Prompt Template Manager
• API Integration Layer
• Evaluation Pipeline
• HHEM-2.1 Evaluation Model
• Results Aggregator
• Metrics Calculator
• Output Processing
• Leaderboard Generator
• Results Database
• Visualization Engine
Integration Layer
• OpenAI Models (GPT-3.5, GPT-4, etc.)
• Anthropic Models (Claude series)
• Google Models (PaLM, Gemini series)
• Other Provider APIs (Cohere, Mistral, etc.)
Core Processing Flow
• Document flow from source → API calls
• Response collection → Evaluation
• Results processing → Leaderboard generation
Component Relationships
• Double lines: API boundaries
• [Input Layer]
• ↓
• [Processing Layer]
• ↓
• [Evaluation Layer]
• ↓
• [Output Layer]
Key Metrics Display
A section showing the main metrics:
• Hallucination Rate
• Factual Consistency Rate
• Answer Rate
• Summary Length
A reference section showing:
• Model versions
• API endpoints
• Prompt templates
• Evaluation criteria
Special Considerations:
HHEM-2.1 Model
• It as a central component
• Role in evaluation
• Connection to the open-source variant
Document Processing
• 831 document subset handling
• Temperature=0 setting
• Content filtering process
Results Processing
• Calculation of metrics
• Visualization pipeline
• Leaderboard generation process