How stanford-crfm/helm works
Purpose and entry points
HELM is a Python framework for reproducible evaluation of language, vision-language, image-generation, and audio-language models. Its primary CLI workflow is benchmark execution, result summarization, and local result serving: src/helm/benchmark/run.py, src/helm/benchmark/presentation/summarize.py, and src/helm/benchmark/server.py. Packaging and CLI definitions are centralized in pyproject.toml.
Run planning and configuration
A run entry is expanded into concrete RunSpec objects that bind a scenario, model deployment, adapter, and metric set. Registries isolate declarative model/tokenizer metadata and benchmark-family configurations from execution logic: src/helm/benchmark/run_expander.py, src/helm/benchmark/run_spec_factory.py, src/helm/benchmark/config_registry.py. Suite-specific run specifications live under src/helm/benchmark/run_specs/, while leaderboard schemas are static YAML under src/helm/benchmark/static/.
Scenario-to-request adaptation
Scenario implementations load/standardize benchmark instances across text, multimodal, audio, medical, and domain-specific datasets. Adapters turn those instances into provider-neutral requests and retain per-request/scenario state. This is the principal boundary between benchmark semantics and model invocation: src/helm/benchmark/scenarios/scenario.py, src/helm/benchmark/adaptation/adapters/adapter_factory.py, src/helm/benchmark/adaptation/request_state.py.
Model execution layer
The runner coordinates requests, caching, concurrency, retries, and persistence of raw outputs; multi-GPU and Slurm execution are alternate execution backends. A unified client interface dispatches to hosted providers or local/Hugging Face/vLLM-style runtimes. Key paths: src/helm/benchmark/runner.py, src/helm/clients/auto_client.py, src/helm/common/cache.py. External services include OpenAI, Anthropic, Google, AWS Bedrock, Cohere, Hugging Face, Together, OpenRouter, and provider-compatible HTTP endpoints.
Evaluation and annotation
Completed request states flow through metrics and, where required, annotation/judge stages to create statistics for accuracy, efficiency, bias, safety, toxicity, and modality-specific quality. Metric services provide shared execution infrastructure; annotator factories select benchmark-specific or model-based evaluators: src/helm/benchmark/metrics/metric_service.py, src/helm/benchmark/annotation_executor.py, src/helm/benchmark/annotation/annotator_factory.py.
Results presentation and serving
Presentation code aggregates run artifacts into tables, schemas, plots, and frontend-consumable result files. The Python server exposes a suite’s results and static assets: src/helm/benchmark/presentation/summarize.py, src/helm/benchmark/presentation/schema.py, src/helm/benchmark/server.py. Data is file-oriented at this boundary; sensitive scenario states can be encrypted/redacted through repository scripts.
Frontend and deployment
A separate React/TypeScript Vite frontend renders leaderboards, runs, requests, predictions, instances, and schemas obtained from benchmark endpoints. Its runtime configuration determines benchmark/release URLs: helm-frontend/src/main.tsx, helm-frontend/src/routes/Run.tsx, helm-frontend/src/services/getRunSpecByName.ts. Built frontend artifacts are vendored for Python serving in src/helm/benchmark/static_build/; GitHub Actions build/test/deploy the frontend and publish the Python package.
Main entry points: runner.py Runner · run_spec_factory.py RunSpec factory · summarize.py Result summarizer · annotation_executor.py Annotation executor
Components 21
Run planning
- Benchmark CLI · Runs benchmarks, summaries, and local serving.
src/helm/benchmark/run.py - Run expander · Expands run entries into concrete run specifications.
src/helm/benchmark/run_expander.py - RunSpec factory · Binds scenarios, deployments, adapters, and metrics.
src/helm/benchmark/run_spec_factory.py - Config registry · Provides declarative model and benchmark-family configuration.
src/helm/benchmark/config_registry.py - Suite run specs
Execution and evaluation
- Scenarios · Load and standardize multimodal benchmark instances.
src/helm/benchmark/scenarios/scenario.py - Adapter factory · Creates provider-neutral requests from scenario instances.
src/helm/benchmark/adaptation/adapters/adapter_factory.py - Request state · Retains request, scenario, and response state.
src/helm/benchmark/adaptation/request_state.py - Runner · Coordinates requests, retries, concurrency, caching, and artifacts.
src/helm/benchmark/runner.py - Cache
src/helm/common/cache.py - Unified model client · Dispatches requests to hosted and local model runtimes.
src/helm/clients/auto_client.py - Model providers · Hosted APIs and provider-compatible local runtimes.
- Annotation executor · Runs required judge and annotation stages.
src/helm/benchmark/annotation_executor.py - Annotator factory
src/helm/benchmark/annotation/annotator_factory.py - Metric service · Computes shared benchmark metrics and statistics.
src/helm/benchmark/metrics/metric_service.py
Results and UI
- Result summarizer · Aggregates run artifacts into result files and tables.
src/helm/benchmark/presentation/summarize.py - Presentation schema
src/helm/benchmark/presentation/schema.py - Results server · Serves suite results and static frontend assets.
src/helm/benchmark/server.py - React frontend · Renders leaderboards, runs, requests, and schemas.
helm-frontend/src/main.tsx - Vendored frontend build · Built frontend artifacts served by Python.
- Leaderboard schemas
Connections 23
- Benchmark CLI → Run expander: expands runs
- Run expander → RunSpec factory: creates specs
- Suite run specs → Run expander: defines suites
- Config registry → RunSpec factory: supplies metadata
- RunSpec factory → Scenarios: selects scenario
- RunSpec factory → Adapter factory: selects adapter
- RunSpec factory → Runner: executes RunSpec
- Scenarios → Adapter factory: benchmark instances
- Adapter factory → Request state: builds requests
- Request state → Runner: request states
- Runner → Cache: reads/writes
- Runner → Unified model client: invokes models
- Unified model client → Model providers: provider requests
- Runner → Annotation executor: completed states
- Annotation executor → Annotator factory: selects annotators
- Annotation executor → Metric service: annotated states
- Runner → Metric service: raw outputs
- Metric service → Result summarizer: run statistics
- Leaderboard schemas → Result summarizer: presentation rules
- Result summarizer → Presentation schema: writes result schema
- Result summarizer → Results server: result artifacts
- Vendored frontend build → Results server: static assets
- Results server → React frontend: benchmark endpoints