# bassrehab/spark-llm-eval architecture

> Architecture diagram of the GitHub repository bassrehab/spark-llm-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bassrehab/spark-llm-eval
- Repository: https://github.com/bassrehab/spark-llm-eval

## Overview

spark-llm-eval is a Spark-native framework for evaluating language models at scale. Its principal workflow is to define an evaluation task, load a Delta-backed dataset, run distributed inference, score predictions with configured metrics, and return statistical results; optional caching and MLflow tracking support cost control and reproducibility. Separate capabilities include agent and RAG evaluation and direct statistical model comparisons. Relationships are based on the README and SOURCE INDEX references; unsampled internals are not assumed to establish additional calls.

Main entry points: [`spark_llm_eval/orchestrator/runner.py`](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/orchestrator/runner.py) (Evaluation runner), [`spark_llm_eval/inference/batch_udf.py`](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/batch_udf.py) (Spark batch inference), [`spark_llm_eval/inference/openai_engine.py`](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/openai_engine.py) (OpenAI engine), [`spark_llm_eval/core/task.py`](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/task.py) (Evaluation task)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_setup["Task and data"]
  node_task["Evaluation task<br/>[task.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_dataset[("Delta datasets<br/>[delta_dataset.py]")]
end

subgraph group_runtime["Inference runtime"]
  node_runner["Evaluation runner<br/>[runner.py]"]
  node_batch["Spark batch inference<br/>[batch_udf.py]"]
  node_engine["Inference interface<br/>[base.py]"]
  node_openai["OpenAI engine<br/>[openai_engine.py]"]
  node_anthropic["Anthropic engine"]
  node_gemini["Gemini engine<br/>[gemini_engine.py]"]
  node_ratelimit["Rate limiting<br/>[rate_limiter.py]"]
end

subgraph group_scoring["Evaluation and analysis"]
  node_metrics["Metric families<br/>[lexical.py]"]
  node_semantic["Semantic metrics<br/>[semantic.py]"]
  node_judge["LLM judge metrics<br/>[llm_judge.py]"]
  node_aggregator["Metric aggregation<br/>[aggregator.py]"]
  node_statistics["Statistical analysis<br/>[confidence.py]"]
  node_result["Evaluation results<br/>[result.py]"]
end

subgraph group_operations["Persistence and tracking"]
  node_cache[("Delta response cache<br/>[delta_cache.py]")]
  node_tracker["MLflow tracking<br/>[mlflow_tracker.py]"]
end

subgraph group_special["Specialized evaluation"]
  node_rag["RAG metrics<br/>[rag.py]"]
  node_agents["Agent evaluation<br/>[trajectory.py]"]
end

node_user(("Evaluator"))
node_providers{{"Model providers"}}
node_delta[("Delta Lake")]
node_mlflow{{"MLflow"}}

node_user -->|"defines"| node_task
node_user -->|"starts run"| node_runner
node_task -->|"configures"| node_runner
node_config -->|"configures"| node_runner
node_dataset -.->|"reads"| node_delta
node_runner -->|"loads"| node_dataset
node_runner -->|"dispatches inference"| node_batch
node_batch -->|"selects provider"| node_openai
node_batch -->|"selects provider"| node_anthropic
node_batch -->|"selects provider"| node_gemini
node_openai -.->|"calls API"| node_providers
node_anthropic -.->|"calls API"| node_providers
node_gemini -.->|"calls API"| node_providers
node_openai -->|"acquires tokens"| node_ratelimit
node_runner -->|"uses cache"| node_cache
node_cache -.->|"stores responses"| node_delta
node_runner -->|"computes scores"| node_metrics
node_runner -->|"computes scores"| node_semantic
node_runner -->|"computes scores"| node_judge
node_runner -->|"computes scores"| node_rag
node_agents -->|"uses metric base"| node_metrics
node_runner -->|"aggregates scores"| node_aggregator
node_runner -->|"computes statistics"| node_statistics
node_runner -->|"returns result"| node_result
node_user -->|"compares models"| node_statistics
node_user -->|"logs experiment"| node_tracker
node_tracker -.->|"logs to"| node_mlflow

click node_task "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/task.py"
click node_config "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/config.py"
click node_dataset "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/datasets/delta_dataset.py"
click node_runner "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/orchestrator/runner.py"
click node_batch "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/batch_udf.py"
click node_engine "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/base.py"
click node_openai "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/openai_engine.py"
click node_anthropic "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/anthropic_engine.py"
click node_gemini "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/gemini_engine.py"
click node_ratelimit "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/rate_limiter.py"
click node_cache "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/cache/delta_cache.py"
click node_metrics "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/lexical.py"
click node_semantic "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/semantic.py"
click node_judge "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/llm_judge.py"
click node_rag "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/rag.py"
click node_agents "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/agents/trajectory.py"
click node_aggregator "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/aggregator.py"
click node_statistics "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/statistics/confidence.py"
click node_result "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/result.py"
click node_tracker "https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/tracking/mlflow_tracker.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task,node_config,node_dataset toneBlue
class node_runner,node_batch,node_engine,node_openai,node_anthropic,node_gemini,node_ratelimit,node_delta toneAmber
class node_metrics,node_semantic,node_judge,node_aggregator,node_statistics,node_result toneMint
class node_cache,node_tracker toneRose
class node_rag,node_agents,node_user,node_providers,node_mlflow toneIndigo
```

## Components

### Task and data

- [Evaluation task](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/task.py): `spark_llm_eval/core/task.py`
- [Run configuration](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/config.py): `spark_llm_eval/core/config.py`
- [Delta datasets](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/datasets/delta_dataset.py): `spark_llm_eval/datasets/delta_dataset.py`

### Inference runtime

- [Evaluation runner](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/orchestrator/runner.py): `spark_llm_eval/orchestrator/runner.py`
- [Spark batch inference](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/batch_udf.py): `spark_llm_eval/inference/batch_udf.py`
- [Inference interface](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/base.py): `spark_llm_eval/inference/base.py`
- [OpenAI engine](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/openai_engine.py): `spark_llm_eval/inference/openai_engine.py`
- [Anthropic engine](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/anthropic_engine.py): `spark_llm_eval/inference/anthropic_engine.py`
- [Gemini engine](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/gemini_engine.py): `spark_llm_eval/inference/gemini_engine.py`
- [Rate limiting](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/inference/rate_limiter.py): `spark_llm_eval/inference/rate_limiter.py`

### Evaluation and analysis

- [Metric families](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/lexical.py): `spark_llm_eval/evaluation/lexical.py`
- [Semantic metrics](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/semantic.py): `spark_llm_eval/evaluation/semantic.py`
- [LLM judge metrics](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/llm_judge.py): `spark_llm_eval/evaluation/llm_judge.py`
- [Metric aggregation](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/aggregator.py): `spark_llm_eval/evaluation/aggregator.py`
- [Statistical analysis](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/statistics/confidence.py): `spark_llm_eval/statistics/confidence.py`
- [Evaluation results](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/core/result.py): `spark_llm_eval/core/result.py`

### Persistence and tracking

- [Delta response cache](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/cache/delta_cache.py): `spark_llm_eval/cache/delta_cache.py`
- [MLflow tracking](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/tracking/mlflow_tracker.py): `spark_llm_eval/tracking/mlflow_tracker.py`

### Specialized evaluation

- [RAG metrics](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/evaluation/rag.py): `spark_llm_eval/evaluation/rag.py`
- [Agent evaluation](https://github.com/bassrehab/spark-llm-eval/blob/main/spark_llm_eval/agents/trajectory.py): `spark_llm_eval/agents/trajectory.py`

### Other components

- **Evaluator**
- **Model providers**
- **Delta Lake**
- **MLflow**

## Connections

- Evaluator → Evaluation task: defines (evidence: `README.md`)
- Evaluator → Evaluation runner: starts run (evidence: `README.md`)
- Evaluation task → Evaluation runner: configures (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Run configuration → Evaluation runner: configures (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Delta datasets → Delta Lake: reads
- Evaluation runner → Delta datasets: loads (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → Spark batch inference: dispatches inference (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Spark batch inference → OpenAI engine: selects provider (evidence: `spark_llm_eval/inference/batch_udf.py`)
- Spark batch inference → Anthropic engine: selects provider (evidence: `spark_llm_eval/inference/batch_udf.py`)
- Spark batch inference → Gemini engine: selects provider (evidence: `spark_llm_eval/inference/batch_udf.py`)
- OpenAI engine → Model providers: calls API
- Anthropic engine → Model providers: calls API
- Gemini engine → Model providers: calls API
- OpenAI engine → Rate limiting: acquires tokens (evidence: `spark_llm_eval/inference/openai_engine.py`)
- Evaluation runner → Delta response cache: uses cache (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Delta response cache → Delta Lake: stores responses
- Evaluation runner → Metric families: computes scores (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → Semantic metrics: computes scores (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → LLM judge metrics: computes scores (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → RAG metrics: computes scores (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Agent evaluation → Metric families: uses metric base (evidence: `spark_llm_eval/agents/trajectory.py`)
- Evaluation runner → Metric aggregation: aggregates scores (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → Statistical analysis: computes statistics (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluation runner → Evaluation results: returns result (evidence: `spark_llm_eval/orchestrator/runner.py`)
- Evaluator → Statistical analysis: compares models (evidence: `README.md`)
- Evaluator → MLflow tracking: logs experiment (evidence: `README.md`)
- MLflow tracking → MLflow: logs to
