How modelscope/evalscope works
EvalScope is a framework for evaluating language and multimodal models, stress-testing inference services, and inspecting results. The principal workflow starts with a benchmark or performance run, invokes a model or backend, scores or measures outputs, and presents reports. Separate branches support agent-loop evaluation and RAG evaluation. README claims establish several product-level links; sampled code confirms the performance pipeline, external-agent bridge, and service registration. Other implementation paths are included where the tree supports them, but their internal call relationships were not sampled.
Main entry points: evaluator.py Evaluation engine · loop.py Agent loop · pipeline.py Request pipeline · App.tsx Web dashboard
Components 23
Model evaluation
- CLI commands
evalscope/cli/cli.py - Run configuration
evalscope/config.py - Benchmark datasets
evalscope/api/dataset/loader.py - Evaluation engine
evalscope/evaluator/evaluator.py - Evaluation metrics
evalscope/metrics - RAG evaluation
evalscope/backend/rag_eval/backend_manager.py - External backends
evalscope/backend
Agent evaluation
- Agent loop
evalscope/api/agent/loop.py - Agent strategies
evalscope/agent/strategies - Agent tools
evalscope/agent/tools - Agent environments
evalscope/agent/environments - External agent bridge
evalscope/agent/external/bridge/server.py
Performance testing
- Performance command
evalscope/cli/start_perf.py - Benchmark runner
evalscope/perf/main.py - Request pipeline
evalscope/perf/core/pipeline.py - API plugins
evalscope/perf/plugin/api/base.py - Performance metrics
evalscope/perf/utils/benchmark_util.py
Results and dashboard
- HTTP service
evalscope/service/app.py - Report generation
evalscope/report/generator.py - Web dashboard
evalscope/web/src/App.tsx
Models and backends
- Model adapters
evalscope/models - Backend interface
evalscope/backend/base.py
Other components
- User
Connections 24
- User → CLI commands: starts run
- CLI commands → Evaluation engine: runs evaluation
- Run configuration → Evaluation engine: configures
- Evaluation engine → Benchmark datasets: uses benchmarks
- Evaluation engine → Evaluation metrics: scores outputs
- Evaluation engine → Model adapters: evaluates models
- Evaluation engine → RAG evaluation: selects backend
- Evaluation engine → External backends: selects backend
- Evaluation engine → Agent loop: runs agent tasks
- Agent loop → Agent strategies: uses strategies
- Agent loop → Agent tools: uses tools
- Agent loop → Agent environments: uses sandbox
- Agent loop → External agent bridge: supports bridge
- User → Performance command: starts test
- Performance command → Benchmark runner: runs benchmark
- Benchmark runner → Request pipeline: runs pipeline
evalscope/perf/main.py - Request pipeline → API plugins: dispatches requests
evalscope/perf/core/pipeline.py - Request pipeline → Performance metrics: consumes results
evalscope/perf/core/pipeline.py - API plugins → Model adapters: requests inference
- Evaluation engine → Report generation: produces reports
- HTTP service → Web dashboard: serves dashboard
evalscope/service/app.py - Web dashboard → HTTP service: calls API
- Report generation → Web dashboard: visualized in
- Backend interface → RAG evaluation: extended by
evalscope/backend/rag_eval/backend_manager.py