How cyberark/simple-llm-eval works
1. Repository purpose
- This is a Python package called simpleval for evaluating LLM outputs using the “LLM as a Judge” pattern.
- It is not just an evaluator: it also contains a CLI, judge/model adapters, metrics computation, report generation, sample eval sets, and a small React-based report frontend.
2. Top-level architecture
- The core runtime is Python under simpleval/.
- The package is organized around three main flows:
1. run an evaluation against an eval set,
2. judge or score the results,
3. render outputs as console or HTML reports.
- Supporting layers provide schema validation, retry/error handling, parallel execution, and provider-specific integrations.
3. CLI and command layer
- simpleval/__main__.py, simpleval/main.py, and simpleval/cli_args.py form the CLI entrypoint.
- simpleval/commands/ contains the user-facing commands:
- run_command.py for executing evals,
- init_command/ for creating new eval-set scaffolding,
- list_models_command.py, metrics_explorer_command.py, judge_explorer_command.py, and litellm_models_explorer_command.py for discovery/help,
- reporting/ for compare, summarize, and eval report generation.
- The CLI is the primary boundary between users and the library internals.
4. Evaluation execution pipeline
- The evaluation engine lives in simpleval/evaluation/.
- eval_runner.py coordinates the main eval workflow, while llm_task_runner.py handles execution of task-specific LLM work.
- parallel_runner/parallel_runner.py provides concurrent execution support, so evals can be run over multiple test cases efficiently.
- This layer consumes eval-set definitions and produces structured result artifacts, typically JSONL-based.
5. Eval-set model and task packaging
- simpleval/eval_sets/ contains bundled example eval sets such as dummy_task, empty, and detect_user_action.
- Each eval set has a clear structure: configuration, ground truth, testcases, and per-testcase handlers.
- simpleval/testcases/ and the test resources under tests/resources/eval_sets/ show the expected packaging contract for custom tasks.
- In architecture terms, eval sets are the domain inputs that the runner loads and executes.
6. Judge subsystem
- simpleval/evaluation/judges/ is a provider abstraction for “LLM as a Judge.”
- base_judge.py defines the shared judge interface, while judge_provider.py and judge_utils.py orchestrate selection and common behavior.
- Provider implementations live under judges/models/ and include adapters for:
- OpenAI,
- Anthropic,
- Azure,
- Google Gemini,
- Vertex AI,
- AWS Bedrock variants,
- LiteLLM structured output,
- and a dummy judge for testing.
- This is the main external-service boundary: the framework delegates scoring/assessment to specific LLM providers through this layer.
7. Metrics subsystem
- simpleval/evaluation/metrics/ contains the metric framework used to score outputs.
- base_metric.py, calc.py, normalizers.py, and the parser modules define the shared metric lifecycle and output interpretation.
- Provider-specific metric implementations are organized similarly to judges, with model families under metrics/models/.
- Metrics include evaluation dimensions such as coherence, correctness, completeness, faithfulness, relevance, helpfulness, readability, and instruction-following.
- Output parsing supports both standard and XML-shaped structured outputs.
8. Schemas, validation, and result shapes
- The repository uses explicit schemas throughout simpleval/evaluation/schemas/, simpleval/testcases/schemas/, and simpleval/validations.py.
- These schemas define the shapes for eval configs, eval cases, results, and task outputs.
- This boundary is important because it separates “task data” from execution logic and keeps reports/metrics consuming stable structures.
9. Reporting architecture
- Reporting lives under simpleval/commands/reporting/.
- There are three main report families:
- compare: compares multiple runs or result sets,
- eval: renders evaluation summaries,
- summarize: produces higher-level LLM summary reports.
- Each report type has both console-oriented and HTML-oriented paths, with shared helpers in reporting/utils.py and eval/html_common.py.
- Reports consume evaluation results and convert them into human-readable analysis artifacts.
10. HTML report frontend
- reports-frontend/ is a separate Node/Vite/React application used to render rich HTML reports.
- It includes components for compare, eval, and summary reports, with tests and build configs for each view.
- The Python side injects placeholders into HTML templates such as compare_report_template.html and llm_eval_report_template.html, then the frontend bundle fills in the report UI.
- This creates a clear split: Python produces report data, the frontend handles presentation.
11. External provider and tooling integration
- The package integrates with multiple LLM vendors and with LiteLLM as a compatibility layer for provider access.
- Several sample scripts under tools/sample_code/ demonstrate provider calls and explorer usage.
- This matters architecturally because the framework is intentionally provider-agnostic, and those adapters are a first-class extension point.
12. Operational utilities
- simpleval/global_config/ manages shared runtime behavior such as retries.
- simpleval/utilities/ contains console formatting, file helpers, retryable/error handling, and supporting logic used across commands and evaluation.
- simpleval/logger.py provides package-level logging behavior.
13. Packaging, docs, and lifecycle support
- The repo is packaged as a Python project via pyproject.toml, with dependency pins in requirements.txt and uv.lock.
- Documentation is built from docs/ with MkDocs.
- The .github/ workflows, ci/, and tools/ directories support testing, releases, changelog/version management, docs publishing, and release tagging.
- These are not runtime subsystems, but they shape how the project is built, validated, and shipped.
14. Testing layout and confidence boundaries
- tests/unit/ covers CLI, evaluation, metrics, parsing, reporting, and initialization behavior.
- tests/integration/ validates judge integrations and cross-provider behaviors.
- Test resources include golden eval sets and provider-specific result fixtures, which strongly suggest the architecture is verified through file-based contracts and deterministic artifacts.
15. End-to-end data flow
- A typical flow is: user invokes the CLI → command loads an eval set/config → task runners execute testcases, possibly in parallel → judges/metrics score outputs → results are normalized into schema-defined artifacts → console and/or HTML reports are generated.
- The main architectural boundaries are therefore:
- CLI vs core engine,
- eval/task data vs execution logic,
- judge/metric provider adapters vs framework abstractions,
- Python report generation vs React presentation.
Main entry points: eval_runner.py Eval runner · eval_report_command.py Reports · test_eval_runner.py Tests · main.py Main
Components 24
CLI
User entrypoints and command dispatch
- Entry · Package entrypoint and CLI bootstrap
simpleval/__main__.py - Main · Main CLI wiring and command dispatch
simpleval/main.py - Args · Command-line argument definitions
simpleval/cli_args.py - Run eval · Primary command for executing evaluations
simpleval/commands/run_command.py - Init · Creates new eval-set scaffolding
simpleval/commands/init_command/init_command.py - Explorers · Model, judge, and metrics explorer commands
simpleval/commands/list_models_command.py
Evaluation
Execution pipeline, schemas, judges, and metrics
- Eval runner · Coordinates eval workflow across task cases
simpleval/evaluation/eval_runner.py - Task runner · Runs task-specific LLM work for eval cases
simpleval/evaluation/llm_task_runner.py - Parallel · Concurrent execution for eval workloads
simpleval/parallel_runner/parallel_runner.py - Eval schemas · Schemas for eval configs, cases, and results
simpleval/evaluation/schemas/eval_result_schema.py - Task schema · Structured testcase result contract
simpleval/testcases/schemas/llm_task_result.py - Judge core · Shared judge interface and provider selection
simpleval/evaluation/judges/base_judge.py - Judge adapters · Provider-specific judge implementations
simpleval/evaluation/judges/models/open_ai/judge.py - Metrics core · Metric lifecycle, calculation, and normalization
simpleval/evaluation/metrics/base_metric.py - Parsers · Standard and XML structured output parsing
simpleval/evaluation/metrics/parsers/output_parsing.py - Metric models · Provider-backed metric implementations
simpleval/evaluation/metrics/models/litellm_structured_output/coherence.py
Reporting
Console and HTML report generation
- Reports · Compare, eval, and summary report commands
simpleval/commands/reporting/eval/eval_report_command.py - Report utils · Shared report formatting and HTML helpers
simpleval/commands/reporting/utils.py
Frontend
React/Vite app for rendered HTML reports
- Report UI · React/Vite frontend for HTML reports
reports-frontend/src/evalMain.jsx
Support
Shared runtime utilities and configuration
- Global config · Shared retry and runtime configuration
simpleval/global_config/retries.py - Utilities · Console, file, retry, and error handling helpers
simpleval/utilities/console.py - Validation · Repository-wide validation and guardrails
simpleval/validations.py
Eval sets
Bundled task definitions and testcase packaging
- Eval sets · Packaged example eval sets and testcase handlers
simpleval/eval_sets/dummy_task/config.json
Tests
Unit, integration, and fixture-based validation
- Tests · Unit and integration coverage with fixtures
tests/unit/test_eval_runner.py
Connections 23
- Entry → Main: boots · Entrypoint hands off to CLI wiring
- Main → Args: parses · CLI bootstrap reads command arguments
- Main → Run eval: dispatches · Top-level CLI dispatches user commands
- Run eval → Eval runner: starts · Run command triggers evaluation execution
- Init → Eval sets: scaffolds · Init command creates eval-set structure
- Eval runner → Task runner: orchestrates · Eval runner delegates case work to task runner
- Eval runner → Parallel: parallelizes · Execution fan-out uses parallel runner
- Eval runner → Eval schemas: loads · Runner consumes schema-defined configs and results
- Task runner → Task schema: emits · Task execution produces testcase-shaped results
- Task runner → Eval sets: consumes · Task handlers come from packaged eval sets
- Eval runner → Judge core: scores · Evaluation can invoke judge abstraction
- Judge core → Judge adapters: adapts · Judge interface is implemented by provider adapters
- Eval runner → Metrics core: computes · Evaluation aggregates metric calculations
- Metrics core → Parsers: parses · Metric framework interprets structured outputs
- Metrics core → Metric models: implements · Metric framework is backed by provider-specific models
- Reports → Report utils: uses · Report commands share formatting and HTML helpers
- Reports → Report UI: renders · HTML report commands target the React frontend bundle
- Reports → Eval runner: reads · Reports consume eval artifacts from execution
- Global config → Utilities: supports · Shared runtime behavior relies on utility helpers
- Validation → Eval schemas: guards · Validation reinforces schema boundaries
- Tests → Eval runner: verifies · Tests cover evaluation workflow behavior
- Tests → Judge adapters: verifies · Integration fixtures cover provider adapters
- Tests → Reports: verifies · Tests cover report generation commands