How vk-ant/llmevalkit works
llmevalkit is a Python library for evaluating LLM outputs, with offline and LLM-judged metrics. The main workflow is a caller supplying a question, answer, and optional context or reference to Evaluator; it runs selected metrics and returns a scored result. Specialized branches cover compliance, document extraction, governance, security, hallucinations, content detection, multimodal evaluation, and ground-truth comparisons. Evaluation history can also be logged and analyzed. Some subsystem links are supported by the evaluator’s indexed imports or README; unsampled implementations are represented at their existing directories.
Main entry points: evaluator.py Evaluation Runner · llm_client.py LLM Client · base.py Judge Metrics · observe Logging And Reports
Components 20
Evaluation Core
- Evaluation Runner
llmevalkit/evaluator.py - Result Models
llmevalkit/models.py - LLM Client
llmevalkit/llm_client.py
Quality And Grounding
- Judge Metrics
llmevalkit/metrics/base.py - Offline Metrics
llmevalkit/metrics/math_metrics.py - Ground Truth
llmevalkit/groundtruth - Hallucination Checks
llmevalkit/hallucination
Risk And Compliance
- Compliance Checks
llmevalkit/compliance - Governance Checks
llmevalkit/governance/frameworks.py - Security Checks
llmevalkit/security
Specialized Evaluation
- Document Evaluation
llmevalkit/doceval - AI Content Detection
llmevalkit/detection/enhanced.py - Multimodal Metrics
llmevalkit/multimodal - Conversation Evaluation
llmevalkit/conversation - Anomaly Detection
llmevalkit/anomaly - Red-Team Evaluation
llmevalkit/redteam
Evaluation History
- Logging And Reports
llmevalkit/observe - Local JSON History
Other components
- Library Caller
- LLM Providers
Connections 18
- Library Caller → Evaluation Runner: submits evaluation
- Evaluation Runner → Judge Metrics: runs judge metrics
llmevalkit/evaluator.py - Evaluation Runner → Offline Metrics: runs offline metrics
llmevalkit/evaluator.py - Evaluation Runner → Ground Truth: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Hallucination Checks: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Compliance Checks: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Governance Checks: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Security Checks: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Document Evaluation: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → AI Content Detection: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Multimodal Metrics: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Conversation Evaluation: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Anomaly Detection: dispatches checks
llmevalkit/evaluator.py - Evaluation Runner → Result Models: returns scores
llmevalkit/evaluator.py - Judge Metrics → LLM Client: requests judgments
llmevalkit/metrics/base.py - LLM Client → LLM Providers: sends requests
llmevalkit/llm_client.py - Evaluation Runner → Logging And Reports: auto-logs results
llmevalkit/evaluator.py - Logging And Reports → Local JSON History: reads and writes
llmevalkit/observe/__init__.py