# vectara/hallucination-leaderboard architecture

> Architecture diagram of the GitHub repository vectara/hallucination-leaderboard, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vectara/hallucination-leaderboard
- Repository: https://github.com/vectara/hallucination-leaderboard

## Overview

Key Components to Include in the System Design:

##### Data Flow Components
- Input Processing System
- Document Source (CNN/Daily Mail Corpus)
- Prompt Template Manager
- API Integration Layer

- Evaluation Pipeline
- HHEM-2.1 Evaluation Model
- Results Aggregator
- Metrics Calculator

- Output Processing
- Leaderboard Generator
- Results Database
- Visualization Engine

##### Integration Layer
- OpenAI Models (GPT-3.5, GPT-4, etc.)
- Anthropic Models (Claude series)
- Google Models (PaLM, Gemini series)
- Other Provider APIs (Cohere, Mistral, etc.)

##### Core Processing Flow
- Document flow from source → API calls
- Response collection → Evaluation
- Results processing → Leaderboard generation

##### Component Relationships
- Double lines: API boundaries

- [Input Layer]
- ↓
- [Processing Layer]
- ↓
- [Evaluation Layer]
- ↓
- [Output Layer]

##### Key Metrics Display
A section showing the main metrics:
- Hallucination Rate
- Factual Consistency Rate
- Answer Rate
- Summary Length

A reference section showing:
- Model versions
- API endpoints
- Prompt templates
- Evaluation criteria

Special Considerations:

##### HHEM-2.1 Model
- It as a central component
- Role in evaluation
- Connection to the open-source variant

##### Document Processing
- 831 document subset handling
- Temperature=0 setting
- Content filtering process

##### Results Processing
- Calculation of metrics
- Visualization pipeline
- Leaderboard generation process

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        DS[Document Source]
        PT[Prompt Template Manager]
        API[API Integration Layer]
    end

    subgraph "Processing & Evaluation"
        HHEM[HHEM-2.1 Evaluation Model]
        RA[Results Aggregator]
        MC[Metrics Calculator]
    end

    subgraph Output
        LG[Leaderboard Generator]
        RDB[(Results Database)]
        VE[Visualization Engine]
        LVS[Legacy Visualization System]
    end

    subgraph "LLM Providers"
        direction TB
        OPENAI[OpenAI Models]
        ANTHROPIC[Anthropic Models]
        GOOGLE[Google Models]
        OTHER[Other Providers]
    end

    subgraph "Key Metrics"
        HR[Hallucination Rate]
        FCR[Factual Consistency]
        AR[Answer Rate]
        SL[Summary Length]
    end

    %% Data Flow
    DS --> PT
    PT --> API
    API --> |API Calls| OPENAI
    API --> |API Calls| ANTHROPIC
    API --> |API Calls| GOOGLE
    API --> |API Calls| OTHER

    OPENAI --> RA
    ANTHROPIC --> RA
    GOOGLE --> RA
    OTHER --> RA

    RA --> HHEM
    HHEM --> MC
    MC --> HR
    MC --> FCR
    MC --> AR
    MC --> SL

    MC --> LG
    LG --> RDB
    RDB --> VE
    RDB --> LVS

    %% Styling
    classDef core fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef external fill:#90EE90,stroke:#333,stroke-width:2px
    classDef processing fill:#FFE5B4,stroke:#333,stroke-width:2px
    classDef storage fill:#E6E6E6,stroke:#333,stroke-width:2px
    classDef metrics fill:#FFB6C1,stroke:#333,stroke-width:2px

    class DS,PT,API core
    class OPENAI,ANTHROPIC,GOOGLE,OTHER external
    class HHEM,RA,MC processing
    class RDB storage
    class HR,FCR,AR,SL metrics
    class LG,VE,LVS core

    %% Click Events
    click VE "https://github.com/vectara/hallucination-leaderboard/blob/main/img/leaderboard_plot.py"
    click LVS "https://github.com/vectara/hallucination-leaderboard/blob/main/img/leaderboard_plot_old.py"

    %% Legend
    subgraph Legend
        C1[Core Components]:::core
        E1[External Services]:::external
        P1[Processing Components]:::processing
        S1[Storage Components]:::storage
        M1[Metrics]:::metrics
    end
```
