# stanford-crfm/helm architecture

> Architecture diagram of the GitHub repository stanford-crfm/helm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanford-crfm/helm
- Repository: https://github.com/stanford-crfm/helm

## Overview

#### Purpose and entry points
HELM is a Python framework for reproducible evaluation of language, vision-language, image-generation, and audio-language models. Its primary CLI workflow is benchmark execution, result summarization, and local result serving: `src/helm/benchmark/run.py`, `src/helm/benchmark/presentation/summarize.py`, and `src/helm/benchmark/server.py`. Packaging and CLI definitions are centralized in `pyproject.toml`.

#### Run planning and configuration
A run entry is expanded into concrete `RunSpec` objects that bind a scenario, model deployment, adapter, and metric set. Registries isolate declarative model/tokenizer metadata and benchmark-family configurations from execution logic: `src/helm/benchmark/run_expander.py`, `src/helm/benchmark/run_spec_factory.py`, `src/helm/benchmark/config_registry.py`. Suite-specific run specifications live under `src/helm/benchmark/run_specs/`, while leaderboard schemas are static YAML under `src/helm/benchmark/static/`.

#### Scenario-to-request adaptation
Scenario implementations load/standardize benchmark instances across text, multimodal, audio, medical, and domain-specific datasets. Adapters turn those instances into provider-neutral requests and retain per-request/scenario state. This is the principal boundary between benchmark semantics and model invocation: `src/helm/benchmark/scenarios/scenario.py`, `src/helm/benchmark/adaptation/adapters/adapter_factory.py`, `src/helm/benchmark/adaptation/request_state.py`.

#### Model execution layer
The runner coordinates requests, caching, concurrency, retries, and persistence of raw outputs; multi-GPU and Slurm execution are alternate execution backends. A unified client interface dispatches to hosted providers or local/Hugging Face/vLLM-style runtimes. Key paths: `src/helm/benchmark/runner.py`, `src/helm/clients/auto_client.py`, `src/helm/common/cache.py`. External services include OpenAI, Anthropic, Google, AWS Bedrock, Cohere, Hugging Face, Together, OpenRouter, and provider-compatible HTTP endpoints.

#### Evaluation and annotation
Completed request states flow through metrics and, where required, annotation/judge stages to create statistics for accuracy, efficiency, bias, safety, toxicity, and modality-specific quality. Metric services provide shared execution infrastructure; annotator factories select benchmark-specific or model-based evaluators: `src/helm/benchmark/metrics/metric_service.py`, `src/helm/benchmark/annotation_executor.py`, `src/helm/benchmark/annotation/annotator_factory.py`.

#### Results presentation and serving
Presentation code aggregates run artifacts into tables, schemas, plots, and frontend-consumable result files. The Python server exposes a suite’s results and static assets: `src/helm/benchmark/presentation/summarize.py`, `src/helm/benchmark/presentation/schema.py`, `src/helm/benchmark/server.py`. Data is file-oriented at this boundary; sensitive scenario states can be encrypted/redacted through repository scripts.

#### Frontend and deployment
A separate React/TypeScript Vite frontend renders leaderboards, runs, requests, predictions, instances, and schemas obtained from benchmark endpoints. Its runtime configuration determines benchmark/release URLs: `helm-frontend/src/main.tsx`, `helm-frontend/src/routes/Run.tsx`, `helm-frontend/src/services/getRunSpecByName.ts`. Built frontend artifacts are vendored for Python serving in `src/helm/benchmark/static_build/`; GitHub Actions build/test/deploy the frontend and publish the Python package.

Main entry points: [`src/helm/benchmark/runner.py`](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/runner.py) (Runner), [`src/helm/benchmark/run_spec_factory.py`](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_spec_factory.py) (RunSpec factory), [`src/helm/benchmark/presentation/summarize.py`](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/summarize.py) (Result summarizer), [`src/helm/benchmark/annotation_executor.py`](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation_executor.py) (Annotation executor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_planning["Run planning"]
  node_cli["Benchmark CLI<br/>Python entry points<br/>[run.py]"]
  node_run_expander["Run expander<br/>planning<br/>[run_expander.py]"]
  node_run_spec_factory["RunSpec factory<br/>planning"]
  node_config_registry["Config registry<br/>[config_registry.py]"]
  node_suite_specs["Suite run specs<br/>benchmark definitions"]
end

subgraph group_execution["Execution and evaluation"]
  node_scenarios["Scenarios<br/>benchmark semantics<br/>[scenario.py]"]
  node_adapter_factory["Adapter factory<br/>adaptation<br/>[adapter_factory.py]"]
  node_request_state["Request state<br/>execution state<br/>[request_state.py]"]
  node_runner["Runner<br/>orchestrator<br/>[runner.py]"]
  node_cache[("Cache<br/>[cache.py]")]
  node_auto_client["Unified model client<br/>provider dispatch<br/>[auto_client.py]"]
  node_providers{{"Model providers<br/>external services"}}
  node_annotation_executor["Annotation executor<br/>evaluation stage"]
  node_annotator_factory["Annotator factory<br/>evaluator selection"]
  node_metric_service["Metric service<br/>evaluation<br/>[metric_service.py]"]
end

subgraph group_results["Results and UI"]
  node_summarize["Result summarizer<br/>presentation pipeline<br/>[summarize.py]"]
  node_presentation_schema["Presentation schema<br/>result schema<br/>[schema.py]"]
  node_server["Results server<br/>Python HTTP server<br/>[server.py]"]
  node_frontend["React frontend<br/>Vite application<br/>[main.tsx]"]
  node_static_build["Vendored frontend build<br/>static assets"]
  node_leaderboard_schemas["Leaderboard schemas<br/>static YAML"]
end

node_cli -->|"expands runs"| node_run_expander
node_run_expander -->|"creates specs"| node_run_spec_factory
node_suite_specs -->|"defines suites"| node_run_expander
node_config_registry -->|"supplies metadata"| node_run_spec_factory
node_run_spec_factory -->|"selects scenario"| node_scenarios
node_run_spec_factory -->|"selects adapter"| node_adapter_factory
node_run_spec_factory -->|"executes RunSpec"| node_runner
node_scenarios -->|"benchmark instances"| node_adapter_factory
node_adapter_factory -->|"builds requests"| node_request_state
node_request_state -->|"request states"| node_runner
node_runner -->|"reads/writes"| node_cache
node_runner -->|"invokes models"| node_auto_client
node_auto_client -->|"provider requests"| node_providers
node_runner -->|"completed states"| node_annotation_executor
node_annotation_executor -->|"selects annotators"| node_annotator_factory
node_annotation_executor -->|"annotated states"| node_metric_service
node_runner -->|"raw outputs"| node_metric_service
node_metric_service -->|"run statistics"| node_summarize
node_leaderboard_schemas -->|"presentation rules"| node_summarize
node_summarize -->|"writes result schema"| node_presentation_schema
node_summarize -->|"result artifacts"| node_server
node_static_build -->|"static assets"| node_server
node_server -->|"benchmark endpoints"| node_frontend

click node_cli "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run.py"
click node_run_expander "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_expander.py"
click node_run_spec_factory "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_spec_factory.py"
click node_config_registry "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/config_registry.py"
click node_scenarios "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/scenario.py"
click node_adapter_factory "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/adaptation/adapters/adapter_factory.py"
click node_request_state "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/adaptation/request_state.py"
click node_runner "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/runner.py"
click node_cache "https://github.com/stanford-crfm/helm/blob/main/src/helm/common/cache.py"
click node_auto_client "https://github.com/stanford-crfm/helm/blob/main/src/helm/clients/auto_client.py"
click node_annotation_executor "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation_executor.py"
click node_annotator_factory "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation/annotator_factory.py"
click node_metric_service "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/metric_service.py"
click node_summarize "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/summarize.py"
click node_presentation_schema "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/schema.py"
click node_server "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/server.py"
click node_frontend "https://github.com/stanford-crfm/helm/blob/main/helm-frontend/src/main.tsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_run_expander,node_run_spec_factory,node_config_registry,node_suite_specs toneBlue
class node_scenarios,node_adapter_factory,node_request_state,node_runner,node_cache,node_auto_client,node_providers,node_annotation_executor,node_annotator_factory,node_metric_service toneAmber
class node_summarize,node_presentation_schema,node_server,node_frontend,node_static_build,node_leaderboard_schemas toneMint
```

## Components

### Run planning

- [Benchmark CLI](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run.py): Python entry points · Runs benchmarks, summaries, and local serving. · `src/helm/benchmark/run.py`
- [Run expander](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_expander.py): planning · Expands run entries into concrete run specifications. · `src/helm/benchmark/run_expander.py`
- [RunSpec factory](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_spec_factory.py): planning · Binds scenarios, deployments, adapters, and metrics. · `src/helm/benchmark/run_spec_factory.py`
- [Config registry](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/config_registry.py): registry · Provides declarative model and benchmark-family configuration. · `src/helm/benchmark/config_registry.py`
- **Suite run specs**: benchmark definitions

### Execution and evaluation

- [Scenarios](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/scenario.py): benchmark semantics · Load and standardize multimodal benchmark instances. · `src/helm/benchmark/scenarios/scenario.py`
- [Adapter factory](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/adaptation/adapters/adapter_factory.py): adaptation · Creates provider-neutral requests from scenario instances. · `src/helm/benchmark/adaptation/adapters/adapter_factory.py`
- [Request state](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/adaptation/request_state.py): execution state · Retains request, scenario, and response state. · `src/helm/benchmark/adaptation/request_state.py`
- [Runner](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/runner.py): orchestrator · Coordinates requests, retries, concurrency, caching, and artifacts. · `src/helm/benchmark/runner.py`
- [Cache](https://github.com/stanford-crfm/helm/blob/main/src/helm/common/cache.py): cache · `src/helm/common/cache.py`
- [Unified model client](https://github.com/stanford-crfm/helm/blob/main/src/helm/clients/auto_client.py): provider dispatch · Dispatches requests to hosted and local model runtimes. · `src/helm/clients/auto_client.py`
- **Model providers**: external services · Hosted APIs and provider-compatible local runtimes.
- [Annotation executor](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation_executor.py): evaluation stage · Runs required judge and annotation stages. · `src/helm/benchmark/annotation_executor.py`
- [Annotator factory](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation/annotator_factory.py): evaluator selection · `src/helm/benchmark/annotation/annotator_factory.py`
- [Metric service](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/metrics/metric_service.py): evaluation · Computes shared benchmark metrics and statistics. · `src/helm/benchmark/metrics/metric_service.py`

### Results and UI

- [Result summarizer](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/summarize.py): presentation pipeline · Aggregates run artifacts into result files and tables. · `src/helm/benchmark/presentation/summarize.py`
- [Presentation schema](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/schema.py): result schema · `src/helm/benchmark/presentation/schema.py`
- [Results server](https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/server.py): Python HTTP server · Serves suite results and static frontend assets. · `src/helm/benchmark/server.py`
- [React frontend](https://github.com/stanford-crfm/helm/blob/main/helm-frontend/src/main.tsx): Vite application · Renders leaderboards, runs, requests, and schemas. · `helm-frontend/src/main.tsx`
- **Vendored frontend build**: static assets · Built frontend artifacts served by Python.
- **Leaderboard schemas**: static YAML

## Connections

- Benchmark CLI → Run expander: expands runs
- Run expander → RunSpec factory: creates specs
- Suite run specs → Run expander: defines suites
- Config registry → RunSpec factory: supplies metadata
- RunSpec factory → Scenarios: selects scenario
- RunSpec factory → Adapter factory: selects adapter
- RunSpec factory → Runner: executes RunSpec
- Scenarios → Adapter factory: benchmark instances
- Adapter factory → Request state: builds requests
- Request state → Runner: request states
- Runner → Cache: reads/writes
- Runner → Unified model client: invokes models
- Unified model client → Model providers: provider requests
- Runner → Annotation executor: completed states
- Annotation executor → Annotator factory: selects annotators
- Annotation executor → Metric service: annotated states
- Runner → Metric service: raw outputs
- Metric service → Result summarizer: run statistics
- Leaderboard schemas → Result summarizer: presentation rules
- Result summarizer → Presentation schema: writes result schema
- Result summarizer → Results server: result artifacts
- Vendored frontend build → Results server: static assets
- Results server → React frontend: benchmark endpoints
