# modelscope/evalscope architecture

> Architecture diagram of the GitHub repository modelscope/evalscope, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modelscope/evalscope
- Repository: https://github.com/modelscope/evalscope

## Overview

EvalScope is a framework for evaluating language and multimodal models, stress-testing inference services, and inspecting results. The principal workflow starts with a benchmark or performance run, invokes a model or backend, scores or measures outputs, and presents reports. Separate branches support agent-loop evaluation and RAG evaluation. README claims establish several product-level links; sampled code confirms the performance pipeline, external-agent bridge, and service registration. Other implementation paths are included where the tree supports them, but their internal call relationships were not sampled.

Main entry points: [`evalscope/evaluator/evaluator.py`](https://github.com/modelscope/evalscope/blob/main/evalscope/evaluator/evaluator.py) (Evaluation engine), [`evalscope/api/agent/loop.py`](https://github.com/modelscope/evalscope/blob/main/evalscope/api/agent/loop.py) (Agent loop), [`evalscope/perf/core/pipeline.py`](https://github.com/modelscope/evalscope/blob/main/evalscope/perf/core/pipeline.py) (Request pipeline), [`evalscope/web/src/App.tsx`](https://github.com/modelscope/evalscope/blob/main/evalscope/web/src/App.tsx) (Web dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Model evaluation"]
  node_cli["CLI commands<br/>[cli.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_datasets[("Benchmark datasets<br/>[loader.py]")]
  node_evaluation_engine["Evaluation engine<br/>[evaluator.py]"]
  node_metrics["Evaluation metrics"]
  node_rag_backend["RAG evaluation<br/>[backend_manager.py]"]
  node_other_backends["External backends"]
end

subgraph group_agent["Agent evaluation"]
  node_agent_loop["Agent loop<br/>[loop.py]"]
  node_agent_strategies["Agent strategies"]
  node_agent_tools["Agent tools"]
  node_agent_env["Agent environments"]
  node_external_bridge["External agent bridge<br/>[server.py]"]
end

subgraph group_performance["Performance testing"]
  node_perf_cli["Performance command<br/>[start_perf.py]"]
  node_perf_main["Benchmark runner<br/>[main.py]"]
  node_perf_pipeline["Request pipeline<br/>[pipeline.py]"]
  node_perf_api["API plugins<br/>[base.py]"]
  node_perf_metrics["Performance metrics<br/>[benchmark_util.py]"]
end

subgraph group_results["Results and dashboard"]
  node_service["HTTP service<br/>[app.py]"]
  node_reports["Report generation<br/>[generator.py]"]
  node_dashboard["Web dashboard<br/>[App.tsx]"]
end

subgraph group_platform["Models and backends"]
  node_models["Model adapters"]
  node_backend_base["Backend interface<br/>[base.py]"]
end

node_user(("User"))

node_user -->|"starts run"| node_cli
node_cli -.->|"runs evaluation"| node_evaluation_engine
node_config -.->|"configures"| node_evaluation_engine
node_evaluation_engine -.->|"uses benchmarks"| node_datasets
node_evaluation_engine -.->|"scores outputs"| node_metrics
node_evaluation_engine -.->|"evaluates models"| node_models
node_evaluation_engine -.->|"selects backend"| node_rag_backend
node_evaluation_engine -.->|"selects backend"| node_other_backends
node_evaluation_engine -.->|"runs agent tasks"| node_agent_loop
node_agent_loop -.->|"uses strategies"| node_agent_strategies
node_agent_loop -.->|"uses tools"| node_agent_tools
node_agent_loop -.->|"uses sandbox"| node_agent_env
node_agent_loop -.->|"supports bridge"| node_external_bridge
node_user -->|"starts test"| node_perf_cli
node_perf_cli -.->|"runs benchmark"| node_perf_main
node_perf_main -->|"runs pipeline"| node_perf_pipeline
node_perf_pipeline -.->|"dispatches requests"| node_perf_api
node_perf_pipeline -->|"consumes results"| node_perf_metrics
node_perf_api -.->|"requests inference"| node_models
node_evaluation_engine -.->|"produces reports"| node_reports
node_service -->|"serves dashboard"| node_dashboard
node_dashboard -.->|"calls API"| node_service
node_reports -.->|"visualized in"| node_dashboard
node_backend_base -->|"extended by"| node_rag_backend

click node_cli "https://github.com/modelscope/evalscope/blob/main/evalscope/cli/cli.py"
click node_config "https://github.com/modelscope/evalscope/blob/main/evalscope/config.py"
click node_datasets "https://github.com/modelscope/evalscope/blob/main/evalscope/api/dataset/loader.py"
click node_evaluation_engine "https://github.com/modelscope/evalscope/blob/main/evalscope/evaluator/evaluator.py"
click node_metrics "https://github.com/modelscope/evalscope/tree/main/evalscope/metrics"
click node_rag_backend "https://github.com/modelscope/evalscope/blob/main/evalscope/backend/rag_eval/backend_manager.py"
click node_other_backends "https://github.com/modelscope/evalscope/tree/main/evalscope/backend"
click node_agent_loop "https://github.com/modelscope/evalscope/blob/main/evalscope/api/agent/loop.py"
click node_agent_strategies "https://github.com/modelscope/evalscope/tree/main/evalscope/agent/strategies"
click node_agent_tools "https://github.com/modelscope/evalscope/tree/main/evalscope/agent/tools"
click node_agent_env "https://github.com/modelscope/evalscope/tree/main/evalscope/agent/environments"
click node_external_bridge "https://github.com/modelscope/evalscope/blob/main/evalscope/agent/external/bridge/server.py"
click node_perf_cli "https://github.com/modelscope/evalscope/blob/main/evalscope/cli/start_perf.py"
click node_perf_main "https://github.com/modelscope/evalscope/blob/main/evalscope/perf/main.py"
click node_perf_pipeline "https://github.com/modelscope/evalscope/blob/main/evalscope/perf/core/pipeline.py"
click node_perf_api "https://github.com/modelscope/evalscope/blob/main/evalscope/perf/plugin/api/base.py"
click node_perf_metrics "https://github.com/modelscope/evalscope/blob/main/evalscope/perf/utils/benchmark_util.py"
click node_models "https://github.com/modelscope/evalscope/tree/main/evalscope/models"
click node_backend_base "https://github.com/modelscope/evalscope/blob/main/evalscope/backend/base.py"
click node_service "https://github.com/modelscope/evalscope/blob/main/evalscope/service/app.py"
click node_reports "https://github.com/modelscope/evalscope/blob/main/evalscope/report/generator.py"
click node_dashboard "https://github.com/modelscope/evalscope/blob/main/evalscope/web/src/App.tsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_datasets,node_evaluation_engine,node_metrics,node_rag_backend,node_other_backends,node_user toneBlue
class node_agent_loop,node_agent_strategies,node_agent_tools,node_agent_env,node_external_bridge toneAmber
class node_perf_cli,node_perf_main,node_perf_pipeline,node_perf_api,node_perf_metrics toneMint
class node_service,node_reports,node_dashboard toneRose
class node_models,node_backend_base toneIndigo
```

## Components

### Model evaluation

- [CLI commands](https://github.com/modelscope/evalscope/blob/main/evalscope/cli/cli.py): `evalscope/cli/cli.py`
- [Run configuration](https://github.com/modelscope/evalscope/blob/main/evalscope/config.py): `evalscope/config.py`
- [Benchmark datasets](https://github.com/modelscope/evalscope/blob/main/evalscope/api/dataset/loader.py): `evalscope/api/dataset/loader.py`
- [Evaluation engine](https://github.com/modelscope/evalscope/blob/main/evalscope/evaluator/evaluator.py): `evalscope/evaluator/evaluator.py`
- [Evaluation metrics](https://github.com/modelscope/evalscope/tree/main/evalscope/metrics): `evalscope/metrics`
- [RAG evaluation](https://github.com/modelscope/evalscope/blob/main/evalscope/backend/rag_eval/backend_manager.py): `evalscope/backend/rag_eval/backend_manager.py`
- [External backends](https://github.com/modelscope/evalscope/tree/main/evalscope/backend): `evalscope/backend`

### Agent evaluation

- [Agent loop](https://github.com/modelscope/evalscope/blob/main/evalscope/api/agent/loop.py): `evalscope/api/agent/loop.py`
- [Agent strategies](https://github.com/modelscope/evalscope/tree/main/evalscope/agent/strategies): `evalscope/agent/strategies`
- [Agent tools](https://github.com/modelscope/evalscope/tree/main/evalscope/agent/tools): `evalscope/agent/tools`
- [Agent environments](https://github.com/modelscope/evalscope/tree/main/evalscope/agent/environments): `evalscope/agent/environments`
- [External agent bridge](https://github.com/modelscope/evalscope/blob/main/evalscope/agent/external/bridge/server.py): `evalscope/agent/external/bridge/server.py`

### Performance testing

- [Performance command](https://github.com/modelscope/evalscope/blob/main/evalscope/cli/start_perf.py): `evalscope/cli/start_perf.py`
- [Benchmark runner](https://github.com/modelscope/evalscope/blob/main/evalscope/perf/main.py): `evalscope/perf/main.py`
- [Request pipeline](https://github.com/modelscope/evalscope/blob/main/evalscope/perf/core/pipeline.py): `evalscope/perf/core/pipeline.py`
- [API plugins](https://github.com/modelscope/evalscope/blob/main/evalscope/perf/plugin/api/base.py): `evalscope/perf/plugin/api/base.py`
- [Performance metrics](https://github.com/modelscope/evalscope/blob/main/evalscope/perf/utils/benchmark_util.py): `evalscope/perf/utils/benchmark_util.py`

### Results and dashboard

- [HTTP service](https://github.com/modelscope/evalscope/blob/main/evalscope/service/app.py): `evalscope/service/app.py`
- [Report generation](https://github.com/modelscope/evalscope/blob/main/evalscope/report/generator.py): `evalscope/report/generator.py`
- [Web dashboard](https://github.com/modelscope/evalscope/blob/main/evalscope/web/src/App.tsx): `evalscope/web/src/App.tsx`

### Models and backends

- [Model adapters](https://github.com/modelscope/evalscope/tree/main/evalscope/models): `evalscope/models`
- [Backend interface](https://github.com/modelscope/evalscope/blob/main/evalscope/backend/base.py): `evalscope/backend/base.py`

### Other components

- **User**

## Connections

- User → CLI commands: starts run
- CLI commands → Evaluation engine: runs evaluation
- Run configuration → Evaluation engine: configures
- Evaluation engine → Benchmark datasets: uses benchmarks
- Evaluation engine → Evaluation metrics: scores outputs
- Evaluation engine → Model adapters: evaluates models
- Evaluation engine → RAG evaluation: selects backend
- Evaluation engine → External backends: selects backend
- Evaluation engine → Agent loop: runs agent tasks
- Agent loop → Agent strategies: uses strategies
- Agent loop → Agent tools: uses tools
- Agent loop → Agent environments: uses sandbox
- Agent loop → External agent bridge: supports bridge
- User → Performance command: starts test
- Performance command → Benchmark runner: runs benchmark
- Benchmark runner → Request pipeline: runs pipeline (evidence: `evalscope/perf/main.py`)
- Request pipeline → API plugins: dispatches requests (evidence: `evalscope/perf/core/pipeline.py`)
- Request pipeline → Performance metrics: consumes results (evidence: `evalscope/perf/core/pipeline.py`)
- API plugins → Model adapters: requests inference
- Evaluation engine → Report generation: produces reports
- HTTP service → Web dashboard: serves dashboard (evidence: `evalscope/service/app.py`)
- Web dashboard → HTTP service: calls API
- Report generation → Web dashboard: visualized in
- Backend interface → RAG evaluation: extended by (evidence: `evalscope/backend/rag_eval/backend_manager.py`)
