# cyberark/simple-llm-eval architecture

> Architecture diagram of the GitHub repository cyberark/simple-llm-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cyberark/simple-llm-eval
- Repository: https://github.com/cyberark/simple-llm-eval

## Overview

1. **Repository purpose**
   - This is a Python package called **simpleval** for evaluating LLM outputs using the “LLM as a Judge” pattern.
   - It is not just an evaluator: it also contains a CLI, judge/model adapters, metrics computation, report generation, sample eval sets, and a small React-based report frontend.

2. **Top-level architecture**
   - The core runtime is Python under `simpleval/`.
   - The package is organized around three main flows:
     1. run an evaluation against an eval set,
     2. judge or score the results,
     3. render outputs as console or HTML reports.
   - Supporting layers provide schema validation, retry/error handling, parallel execution, and provider-specific integrations.

3. **CLI and command layer**
   - `simpleval/__main__.py`, `simpleval/main.py`, and `simpleval/cli_args.py` form the CLI entrypoint.
   - `simpleval/commands/` contains the user-facing commands:
     - `run_command.py` for executing evals,
     - `init_command/` for creating new eval-set scaffolding,
     - `list_models_command.py`, `metrics_explorer_command.py`, `judge_explorer_command.py`, and `litellm_models_explorer_command.py` for discovery/help,
     - `reporting/` for compare, summarize, and eval report generation.
   - The CLI is the primary boundary between users and the library internals.

4. **Evaluation execution pipeline**
   - The evaluation engine lives in `simpleval/evaluation/`.
   - `eval_runner.py` coordinates the main eval workflow, while `llm_task_runner.py` handles execution of task-specific LLM work.
   - `parallel_runner/parallel_runner.py` provides concurrent execution support, so evals can be run over multiple test cases efficiently.
   - This layer consumes eval-set definitions and produces structured result artifacts, typically JSONL-based.

5. **Eval-set model and task packaging**
   - `simpleval/eval_sets/` contains bundled example eval sets such as `dummy_task`, `empty`, and `detect_user_action`.
   - Each eval set has a clear structure: configuration, ground truth, testcases, and per-testcase handlers.
   - `simpleval/testcases/` and the test resources under `tests/resources/eval_sets/` show the expected packaging contract for custom tasks.
   - In architecture terms, eval sets are the domain inputs that the runner loads and executes.

6. **Judge subsystem**
   - `simpleval/evaluation/judges/` is a provider abstraction for “LLM as a Judge.”
   - `base_judge.py` defines the shared judge interface, while `judge_provider.py` and `judge_utils.py` orchestrate selection and common behavior.
   - Provider implementations live under `judges/models/` and include adapters for:
     - OpenAI,
     - Anthropic,
     - Azure,
     - Google Gemini,
     - Vertex AI,
     - AWS Bedrock variants,
     - LiteLLM structured output,
     - and a dummy judge for testing.
   - This is the main external-service boundary: the framework delegates scoring/assessment to specific LLM providers through this layer.

7. **Metrics subsystem**
   - `simpleval/evaluation/metrics/` contains the metric framework used to score outputs.
   - `base_metric.py`, `calc.py`, `normalizers.py`, and the parser modules define the shared metric lifecycle and output interpretation.
   - Provider-specific metric implementations are organized similarly to judges, with model families under `metrics/models/`.
   - Metrics include evaluation dimensions such as coherence, correctness, completeness, faithfulness, relevance, helpfulness, readability, and instruction-following.
   - Output parsing supports both standard and XML-shaped structured outputs.

8. **Schemas, validation, and result shapes**
   - The repository uses explicit schemas throughout `simpleval/evaluation/schemas/`, `simpleval/testcases/schemas/`, and `simpleval/validations.py`.
   - These schemas define the shapes for eval configs, eval cases, results, and task outputs.
   - This boundary is important because it separates “task data” from execution logic and keeps reports/metrics consuming stable structures.

9. **Reporting architecture**
   - Reporting lives under `simpleval/commands/reporting/`.
   - There are three main report families:
     - **compare**: compares multiple runs or result sets,
     - **eval**: renders evaluation summaries,
     - **summarize**: produces higher-level LLM summary reports.
   - Each report type has both console-oriented and HTML-oriented paths, with shared helpers in `reporting/utils.py` and `eval/html_common.py`.
   - Reports consume evaluation results and convert them into human-readable analysis artifacts.

10. **HTML report frontend**
   - `reports-frontend/` is a separate Node/Vite/React application used to render rich HTML reports.
   - It includes components for compare, eval, and summary reports, with tests and build configs for each view.
   - The Python side injects placeholders into HTML templates such as `compare_report_template.html` and `llm_eval_report_template.html`, then the frontend bundle fills in the report UI.
   - This creates a clear split: Python produces report data, the frontend handles presentation.

11. **External provider and tooling integration**
   - The package integrates with multiple LLM vendors and with **LiteLLM** as a compatibility layer for provider access.
   - Several sample scripts under `tools/sample_code/` demonstrate provider calls and explorer usage.
   - This matters architecturally because the framework is intentionally provider-agnostic, and those adapters are a first-class extension point.

12. **Operational utilities**
   - `simpleval/global_config/` manages shared runtime behavior such as retries.
   - `simpleval/utilities/` contains console formatting, file helpers, retryable/error handling, and supporting logic used across commands and evaluation.
   - `simpleval/logger.py` provides package-level logging behavior.

13. **Packaging, docs, and lifecycle support**
   - The repo is packaged as a Python project via `pyproject.toml`, with dependency pins in `requirements.txt` and `uv.lock`.
   - Documentation is built from `docs/` with MkDocs.
   - The `.github/` workflows, `ci/`, and `tools/` directories support testing, releases, changelog/version management, docs publishing, and release tagging.
   - These are not runtime subsystems, but they shape how the project is built, validated, and shipped.

14. **Testing layout and confidence boundaries**
   - `tests/unit/` covers CLI, evaluation, metrics, parsing, reporting, and initialization behavior.
   - `tests/integration/` validates judge integrations and cross-provider behaviors.
   - Test resources include golden eval sets and provider-specific result fixtures, which strongly suggest the architecture is verified through file-based contracts and deterministic artifacts.

15. **End-to-end data flow**
   - A typical flow is: user invokes the CLI → command loads an eval set/config → task runners execute testcases, possibly in parallel → judges/metrics score outputs → results are normalized into schema-defined artifacts → console and/or HTML reports are generated.
   - The main architectural boundaries are therefore:
     - CLI vs core engine,
     - eval/task data vs execution logic,
     - judge/metric provider adapters vs framework abstractions,
     - Python report generation vs React presentation.

Main entry points: [`simpleval/evaluation/eval_runner.py`](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/eval_runner.py) (Eval runner), [`simpleval/commands/reporting/eval/eval_report_command.py`](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/reporting/eval/eval_report_command.py) (Reports), [`tests/unit/test_eval_runner.py`](https://github.com/cyberark/simple-llm-eval/blob/main/tests/unit/test_eval_runner.py) (Tests), [`simpleval/main.py`](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/main.py) (Main)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_cli["CLI"]
  node_node_main["Entry<br/>[__main__.py]"]
  node_node_main_mod["Main<br/>cli bootstrap<br/>[main.py]"]
  node_node_cli_args["Args<br/>arg parsing<br/>[cli_args.py]"]
  node_node_run_cmd["Run eval<br/>eval command<br/>[run_command.py]"]
  node_node_init_cmd["Init<br/>scaffold command<br/>[init_command.py]"]
  node_node_discovery_cmds["Explorers<br/>discovery commands"]
end

subgraph group_grp_eval["Evaluation"]
  node_node_eval_runner["Eval runner<br/>execution orchestrator<br/>[eval_runner.py]"]
  node_node_task_runner["Task runner<br/>llm task execution<br/>[llm_task_runner.py]"]
  node_node_parallel["Parallel<br/>concurrency layer<br/>[parallel_runner.py]"]
  node_node_eval_schemas["Eval schemas<br/>schema layer"]
  node_node_testcase_schema["Task schema<br/>task contract<br/>[llm_task_result.py]"]
  node_node_judge_core["Judge core<br/>judge abstraction<br/>[base_judge.py]"]
  node_node_judge_models["Judge adapters<br/>provider adapters<br/>[judge.py]"]
  node_node_metrics_core["Metrics core<br/>metric framework<br/>[base_metric.py]"]
  node_node_metrics_parsers["Parsers<br/>output parsing<br/>[output_parsing.py]"]
  node_node_metrics_models["Metric models<br/>metric adapters<br/>[coherence.py]"]
end

subgraph group_grp_reports["Reporting"]
  node_node_report_cmds["Reports<br/>report commands"]
  node_node_report_utils["Report utils<br/>report helpers<br/>[utils.py]"]
end

subgraph group_grp_frontend["Frontend"]
  node_node_frontend["Report UI<br/>react app<br/>[evalMain.jsx]"]
end

subgraph group_grp_support["Support"]
  node_node_global["Global config<br/>runtime config<br/>[retries.py]"]
  node_node_utils["Utilities<br/>shared helpers<br/>[console.py]"]
  node_node_validation["Validation<br/>[validations.py]"]
end

subgraph group_grp_evalsets["Eval sets"]
  node_node_eval_sets["Eval sets<br/>bundled tasks<br/>[config.json]"]
end

subgraph group_grp_tests["Tests"]
  node_node_tests["Tests<br/>test suite"]
end

node_node_main -->|"boots"| node_node_main_mod
node_node_main_mod -->|"parses"| node_node_cli_args
node_node_main_mod -->|"dispatches"| node_node_run_cmd
node_node_run_cmd -->|"starts"| node_node_eval_runner
node_node_init_cmd -->|"scaffolds"| node_node_eval_sets
node_node_eval_runner -->|"orchestrates"| node_node_task_runner
node_node_eval_runner -->|"parallelizes"| node_node_parallel
node_node_eval_runner -->|"loads"| node_node_eval_schemas
node_node_task_runner -->|"emits"| node_node_testcase_schema
node_node_task_runner -->|"consumes"| node_node_eval_sets
node_node_eval_runner -->|"scores"| node_node_judge_core
node_node_judge_core -->|"adapts"| node_node_judge_models
node_node_eval_runner -->|"computes"| node_node_metrics_core
node_node_metrics_core -->|"parses"| node_node_metrics_parsers
node_node_metrics_core -->|"implements"| node_node_metrics_models
node_node_report_cmds -->|"uses"| node_node_report_utils
node_node_report_cmds -->|"renders"| node_node_frontend
node_node_report_cmds -->|"reads"| node_node_eval_runner
node_node_global -->|"supports"| node_node_utils
node_node_validation -->|"guards"| node_node_eval_schemas
node_node_tests -.->|"verifies"| node_node_eval_runner
node_node_tests -.->|"verifies"| node_node_judge_models
node_node_tests -.->|"verifies"| node_node_report_cmds

click node_node_main "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/__main__.py"
click node_node_main_mod "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/main.py"
click node_node_cli_args "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/cli_args.py"
click node_node_run_cmd "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/run_command.py"
click node_node_init_cmd "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/init_command/init_command.py"
click node_node_discovery_cmds "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/list_models_command.py"
click node_node_eval_runner "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/eval_runner.py"
click node_node_task_runner "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/llm_task_runner.py"
click node_node_parallel "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/parallel_runner/parallel_runner.py"
click node_node_eval_schemas "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/schemas/eval_result_schema.py"
click node_node_testcase_schema "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/testcases/schemas/llm_task_result.py"
click node_node_judge_core "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/judges/base_judge.py"
click node_node_judge_models "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/judges/models/open_ai/judge.py"
click node_node_metrics_core "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/base_metric.py"
click node_node_metrics_parsers "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/parsers/output_parsing.py"
click node_node_metrics_models "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/models/litellm_structured_output/coherence.py"
click node_node_report_cmds "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/reporting/eval/eval_report_command.py"
click node_node_report_utils "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/reporting/utils.py"
click node_node_frontend "https://github.com/cyberark/simple-llm-eval/blob/main/reports-frontend/src/evalMain.jsx"
click node_node_eval_sets "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/eval_sets/dummy_task/config.json"
click node_node_global "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/global_config/retries.py"
click node_node_utils "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/utilities/console.py"
click node_node_validation "https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/validations.py"
click node_node_tests "https://github.com/cyberark/simple-llm-eval/blob/main/tests/unit/test_eval_runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_main,node_node_main_mod,node_node_cli_args,node_node_run_cmd,node_node_init_cmd,node_node_discovery_cmds,node_node_tests toneBlue
class node_node_eval_runner,node_node_task_runner,node_node_parallel,node_node_eval_schemas,node_node_testcase_schema,node_node_judge_core,node_node_judge_models,node_node_metrics_core,node_node_metrics_parsers,node_node_metrics_models toneAmber
class node_node_report_cmds,node_node_report_utils toneMint
class node_node_frontend toneRose
class node_node_global,node_node_utils,node_node_validation toneIndigo
class node_node_eval_sets toneTeal
```

## Components

### CLI

User entrypoints and command dispatch

- [Entry](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/__main__.py): python entrypoint · Package entrypoint and CLI bootstrap · `simpleval/__main__.py`
- [Main](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/main.py): cli bootstrap · Main CLI wiring and command dispatch · `simpleval/main.py`
- [Args](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/cli_args.py): arg parsing · Command-line argument definitions · `simpleval/cli_args.py`
- [Run eval](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/run_command.py): eval command · Primary command for executing evaluations · `simpleval/commands/run_command.py`
- [Init](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/init_command/init_command.py): scaffold command · Creates new eval-set scaffolding · `simpleval/commands/init_command/init_command.py`
- [Explorers](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/list_models_command.py): discovery commands · Model, judge, and metrics explorer commands · `simpleval/commands/list_models_command.py`

### Evaluation

Execution pipeline, schemas, judges, and metrics

- [Eval runner](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/eval_runner.py): execution orchestrator · Coordinates eval workflow across task cases · `simpleval/evaluation/eval_runner.py`
- [Task runner](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/llm_task_runner.py): llm task execution · Runs task-specific LLM work for eval cases · `simpleval/evaluation/llm_task_runner.py`
- [Parallel](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/parallel_runner/parallel_runner.py): concurrency layer · Concurrent execution for eval workloads · `simpleval/parallel_runner/parallel_runner.py`
- [Eval schemas](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/schemas/eval_result_schema.py): schema layer · Schemas for eval configs, cases, and results · `simpleval/evaluation/schemas/eval_result_schema.py`
- [Task schema](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/testcases/schemas/llm_task_result.py): task contract · Structured testcase result contract · `simpleval/testcases/schemas/llm_task_result.py`
- [Judge core](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/judges/base_judge.py): judge abstraction · Shared judge interface and provider selection · `simpleval/evaluation/judges/base_judge.py`
- [Judge adapters](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/judges/models/open_ai/judge.py): provider adapters · Provider-specific judge implementations · `simpleval/evaluation/judges/models/open_ai/judge.py`
- [Metrics core](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/base_metric.py): metric framework · Metric lifecycle, calculation, and normalization · `simpleval/evaluation/metrics/base_metric.py`
- [Parsers](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/parsers/output_parsing.py): output parsing · Standard and XML structured output parsing · `simpleval/evaluation/metrics/parsers/output_parsing.py`
- [Metric models](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/evaluation/metrics/models/litellm_structured_output/coherence.py): metric adapters · Provider-backed metric implementations · `simpleval/evaluation/metrics/models/litellm_structured_output/coherence.py`

### Reporting

Console and HTML report generation

- [Reports](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/reporting/eval/eval_report_command.py): report commands · Compare, eval, and summary report commands · `simpleval/commands/reporting/eval/eval_report_command.py`
- [Report utils](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/commands/reporting/utils.py): report helpers · Shared report formatting and HTML helpers · `simpleval/commands/reporting/utils.py`

### Frontend

React/Vite app for rendered HTML reports

- [Report UI](https://github.com/cyberark/simple-llm-eval/blob/main/reports-frontend/src/evalMain.jsx): react app · React/Vite frontend for HTML reports · `reports-frontend/src/evalMain.jsx`

### Support

Shared runtime utilities and configuration

- [Global config](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/global_config/retries.py): runtime config · Shared retry and runtime configuration · `simpleval/global_config/retries.py`
- [Utilities](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/utilities/console.py): shared helpers · Console, file, retry, and error handling helpers · `simpleval/utilities/console.py`
- [Validation](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/validations.py): input validation · Repository-wide validation and guardrails · `simpleval/validations.py`

### Eval sets

Bundled task definitions and testcase packaging

- [Eval sets](https://github.com/cyberark/simple-llm-eval/blob/main/simpleval/eval_sets/dummy_task/config.json): bundled tasks · Packaged example eval sets and testcase handlers · `simpleval/eval_sets/dummy_task/config.json`

### Tests

Unit, integration, and fixture-based validation

- [Tests](https://github.com/cyberark/simple-llm-eval/blob/main/tests/unit/test_eval_runner.py): test suite · Unit and integration coverage with fixtures · `tests/unit/test_eval_runner.py`

## Connections

- Entry → Main: boots · Entrypoint hands off to CLI wiring
- Main → Args: parses · CLI bootstrap reads command arguments
- Main → Run eval: dispatches · Top-level CLI dispatches user commands
- Run eval → Eval runner: starts · Run command triggers evaluation execution
- Init → Eval sets: scaffolds · Init command creates eval-set structure
- Eval runner → Task runner: orchestrates · Eval runner delegates case work to task runner
- Eval runner → Parallel: parallelizes · Execution fan-out uses parallel runner
- Eval runner → Eval schemas: loads · Runner consumes schema-defined configs and results
- Task runner → Task schema: emits · Task execution produces testcase-shaped results
- Task runner → Eval sets: consumes · Task handlers come from packaged eval sets
- Eval runner → Judge core: scores · Evaluation can invoke judge abstraction
- Judge core → Judge adapters: adapts · Judge interface is implemented by provider adapters
- Eval runner → Metrics core: computes · Evaluation aggregates metric calculations
- Metrics core → Parsers: parses · Metric framework interprets structured outputs
- Metrics core → Metric models: implements · Metric framework is backed by provider-specific models
- Reports → Report utils: uses · Report commands share formatting and HTML helpers
- Reports → Report UI: renders · HTML report commands target the React frontend bundle
- Reports → Eval runner: reads · Reports consume eval artifacts from execution
- Global config → Utilities: supports · Shared runtime behavior relies on utility helpers
- Validation → Eval schemas: guards · Validation reinforces schema boundaries
- Tests → Eval runner: verifies · Tests cover evaluation workflow behavior
- Tests → Judge adapters: verifies · Integration fixtures cover provider adapters
- Tests → Reports: verifies · Tests cover report generation commands