# vk-ant/llmevalkit architecture

> Architecture diagram of the GitHub repository vk-ant/llmevalkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vk-ant/llmevalkit
- Repository: https://github.com/vk-ant/llmevalkit

## Overview

llmevalkit is a Python library for evaluating LLM outputs, with offline and LLM-judged metrics. The main workflow is a caller supplying a question, answer, and optional context or reference to Evaluator; it runs selected metrics and returns a scored result. Specialized branches cover compliance, document extraction, governance, security, hallucinations, content detection, multimodal evaluation, and ground-truth comparisons. Evaluation history can also be logged and analyzed. Some subsystem links are supported by the evaluator’s indexed imports or README; unsampled implementations are represented at their existing directories.

Main entry points: [`llmevalkit/evaluator.py`](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/evaluator.py) (Evaluation Runner), [`llmevalkit/llm_client.py`](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/llm_client.py) (LLM Client), [`llmevalkit/metrics/base.py`](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/metrics/base.py) (Judge Metrics), [`llmevalkit/observe`](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/observe) (Logging And Reports)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Evaluation Core"]
  node_evaluator["Evaluation Runner<br/>[evaluator.py]"]
  node_result["Result Models<br/>[models.py]"]
  node_llm["LLM Client<br/>[llm_client.py]"]
end

subgraph group_quality["Quality And Grounding"]
  node_base_metrics["Judge Metrics<br/>[base.py]"]
  node_math_metrics["Offline Metrics<br/>[math_metrics.py]"]
  node_groundtruth["Ground Truth"]
  node_hallucination["Hallucination Checks"]
end

subgraph group_assurance["Risk And Compliance"]
  node_compliance["Compliance Checks"]
  node_governance["Governance Checks<br/>[frameworks.py]"]
  node_security["Security Checks"]
end

subgraph group_specialized["Specialized Evaluation"]
  node_document["Document Evaluation"]
  node_detection["AI Content Detection<br/>[enhanced.py]"]
  node_multimodal["Multimodal Metrics"]
  node_conversation["Conversation Evaluation"]
  node_anomaly["Anomaly Detection"]
  node_redteam["Red-Team Evaluation"]
end

subgraph group_operations["Evaluation History"]
  node_observe["Logging And Reports"]
  node_json_logs[("Local JSON History")]
end

node_caller(("Library Caller"))
node_llm_providers["LLM Providers"]

node_caller -->|"submits evaluation"| node_evaluator
node_evaluator -->|"runs judge metrics"| node_base_metrics
node_evaluator -->|"runs offline metrics"| node_math_metrics
node_evaluator -->|"dispatches checks"| node_groundtruth
node_evaluator -->|"dispatches checks"| node_hallucination
node_evaluator -->|"dispatches checks"| node_compliance
node_evaluator -->|"dispatches checks"| node_governance
node_evaluator -->|"dispatches checks"| node_security
node_evaluator -->|"dispatches checks"| node_document
node_evaluator -->|"dispatches checks"| node_detection
node_evaluator -->|"dispatches checks"| node_multimodal
node_evaluator -->|"dispatches checks"| node_conversation
node_evaluator -->|"dispatches checks"| node_anomaly
node_evaluator -->|"returns scores"| node_result
node_base_metrics -->|"requests judgments"| node_llm
node_llm -->|"sends requests"| node_llm_providers
node_evaluator -.->|"auto-logs results"| node_observe
node_observe -->|"reads and writes"| node_json_logs

click node_evaluator "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/evaluator.py"
click node_result "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/models.py"
click node_llm "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/llm_client.py"
click node_base_metrics "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/metrics/base.py"
click node_math_metrics "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/metrics/math_metrics.py"
click node_groundtruth "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/groundtruth"
click node_hallucination "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/hallucination"
click node_compliance "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/compliance"
click node_governance "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/governance/frameworks.py"
click node_security "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/security"
click node_document "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/doceval"
click node_detection "https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/detection/enhanced.py"
click node_multimodal "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/multimodal"
click node_conversation "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/conversation"
click node_anomaly "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/anomaly"
click node_redteam "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/redteam"
click node_observe "https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/observe"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_evaluator,node_result,node_llm toneBlue
class node_base_metrics,node_math_metrics,node_groundtruth,node_hallucination toneAmber
class node_compliance,node_governance,node_security toneMint
class node_document,node_detection,node_multimodal,node_conversation,node_anomaly,node_redteam toneRose
class node_observe,node_json_logs,node_caller,node_llm_providers toneIndigo
```

## Components

### Evaluation Core

- [Evaluation Runner](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/evaluator.py): `llmevalkit/evaluator.py`
- [Result Models](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/models.py): `llmevalkit/models.py`
- [LLM Client](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/llm_client.py): `llmevalkit/llm_client.py`

### Quality And Grounding

- [Judge Metrics](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/metrics/base.py): `llmevalkit/metrics/base.py`
- [Offline Metrics](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/metrics/math_metrics.py): `llmevalkit/metrics/math_metrics.py`
- [Ground Truth](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/groundtruth): `llmevalkit/groundtruth`
- [Hallucination Checks](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/hallucination): `llmevalkit/hallucination`

### Risk And Compliance

- [Compliance Checks](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/compliance): `llmevalkit/compliance`
- [Governance Checks](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/governance/frameworks.py): `llmevalkit/governance/frameworks.py`
- [Security Checks](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/security): `llmevalkit/security`

### Specialized Evaluation

- [Document Evaluation](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/doceval): `llmevalkit/doceval`
- [AI Content Detection](https://github.com/vk-ant/llmevalkit/blob/main/llmevalkit/detection/enhanced.py): `llmevalkit/detection/enhanced.py`
- [Multimodal Metrics](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/multimodal): `llmevalkit/multimodal`
- [Conversation Evaluation](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/conversation): `llmevalkit/conversation`
- [Anomaly Detection](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/anomaly): `llmevalkit/anomaly`
- [Red-Team Evaluation](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/redteam): `llmevalkit/redteam`

### Evaluation History

- [Logging And Reports](https://github.com/vk-ant/llmevalkit/tree/main/llmevalkit/observe): `llmevalkit/observe`
- **Local JSON History**

### Other components

- **Library Caller**
- **LLM Providers**

## Connections

- Library Caller → Evaluation Runner: submits evaluation
- Evaluation Runner → Judge Metrics: runs judge metrics (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Offline Metrics: runs offline metrics (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Ground Truth: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Hallucination Checks: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Compliance Checks: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Governance Checks: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Security Checks: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Document Evaluation: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → AI Content Detection: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Multimodal Metrics: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Conversation Evaluation: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Anomaly Detection: dispatches checks (evidence: `llmevalkit/evaluator.py`)
- Evaluation Runner → Result Models: returns scores (evidence: `llmevalkit/evaluator.py`)
- Judge Metrics → LLM Client: requests judgments (evidence: `llmevalkit/metrics/base.py`)
- LLM Client → LLM Providers: sends requests (evidence: `llmevalkit/llm_client.py`)
- Evaluation Runner → Logging And Reports: auto-logs results (evidence: `llmevalkit/evaluator.py`)
- Logging And Reports → Local JSON History: reads and writes (evidence: `llmevalkit/observe/__init__.py`)
