How bassrehab/spark-llm-eval works
spark-llm-eval is a Spark-native framework for evaluating language models at scale. Its principal workflow is to define an evaluation task, load a Delta-backed dataset, run distributed inference, score predictions with configured metrics, and return statistical results; optional caching and MLflow tracking support cost control and reproducibility. Separate capabilities include agent and RAG evaluation and direct statistical model comparisons. Relationships are based on the README and SOURCE INDEX references; unsampled internals are not assumed to establish additional calls.
Main entry points: runner.py Evaluation runner · batch_udf.py Spark batch inference · openai_engine.py OpenAI engine · task.py Evaluation task
Components 24
Task and data
- Evaluation task
spark_llm_eval/core/task.py - Run configuration
spark_llm_eval/core/config.py - Delta datasets
spark_llm_eval/datasets/delta_dataset.py
Inference runtime
- Evaluation runner
spark_llm_eval/orchestrator/runner.py - Spark batch inference
spark_llm_eval/inference/batch_udf.py - Inference interface
spark_llm_eval/inference/base.py - OpenAI engine
spark_llm_eval/inference/openai_engine.py - Anthropic engine
spark_llm_eval/inference/anthropic_engine.py - Gemini engine
spark_llm_eval/inference/gemini_engine.py - Rate limiting
spark_llm_eval/inference/rate_limiter.py
Evaluation and analysis
- Metric families
spark_llm_eval/evaluation/lexical.py - Semantic metrics
spark_llm_eval/evaluation/semantic.py - LLM judge metrics
spark_llm_eval/evaluation/llm_judge.py - Metric aggregation
spark_llm_eval/evaluation/aggregator.py - Statistical analysis
spark_llm_eval/statistics/confidence.py - Evaluation results
spark_llm_eval/core/result.py
Persistence and tracking
- Delta response cache
spark_llm_eval/cache/delta_cache.py - MLflow tracking
spark_llm_eval/tracking/mlflow_tracker.py
Specialized evaluation
- RAG metrics
spark_llm_eval/evaluation/rag.py - Agent evaluation
spark_llm_eval/agents/trajectory.py
Other components
- Evaluator
- Model providers
- Delta Lake
- MLflow
Connections 27
- Evaluator → Evaluation task: defines
README.md - Evaluator → Evaluation runner: starts run
README.md - Evaluation task → Evaluation runner: configures
spark_llm_eval/orchestrator/runner.py - Run configuration → Evaluation runner: configures
spark_llm_eval/orchestrator/runner.py - Delta datasets → Delta Lake: reads
- Evaluation runner → Delta datasets: loads
spark_llm_eval/orchestrator/runner.py - Evaluation runner → Spark batch inference: dispatches inference
spark_llm_eval/orchestrator/runner.py - Spark batch inference → OpenAI engine: selects provider
spark_llm_eval/inference/batch_udf.py - Spark batch inference → Anthropic engine: selects provider
spark_llm_eval/inference/batch_udf.py - Spark batch inference → Gemini engine: selects provider
spark_llm_eval/inference/batch_udf.py - OpenAI engine → Model providers: calls API
- Anthropic engine → Model providers: calls API
- Gemini engine → Model providers: calls API
- OpenAI engine → Rate limiting: acquires tokens
spark_llm_eval/inference/openai_engine.py - Evaluation runner → Delta response cache: uses cache
spark_llm_eval/orchestrator/runner.py - Delta response cache → Delta Lake: stores responses
- Evaluation runner → Metric families: computes scores
spark_llm_eval/orchestrator/runner.py - Evaluation runner → Semantic metrics: computes scores
spark_llm_eval/orchestrator/runner.py - Evaluation runner → LLM judge metrics: computes scores
spark_llm_eval/orchestrator/runner.py - Evaluation runner → RAG metrics: computes scores
spark_llm_eval/orchestrator/runner.py - Agent evaluation → Metric families: uses metric base
spark_llm_eval/agents/trajectory.py - Evaluation runner → Metric aggregation: aggregates scores
spark_llm_eval/orchestrator/runner.py - Evaluation runner → Statistical analysis: computes statistics
spark_llm_eval/orchestrator/runner.py - Evaluation runner → Evaluation results: returns result
spark_llm_eval/orchestrator/runner.py - Evaluator → Statistical analysis: compares models
README.md - Evaluator → MLflow tracking: logs experiment
README.md - MLflow tracking → MLflow: logs to