How vii1nonly/mcp-eval-harness works
mcp-eval is a reliability evaluation harness for MCP tool servers. A user selects a YAML suite and server command through the CLI; the suite loader creates test cases, the runner executes them through a stdio MCP connector, graders assign pass, fail, or inconclusive verdicts, and the CLI presents a scorecard. The CLI also supports tool discovery, while reports can be written as JSON. The README documents planned recovery, HTTP transport, security/fuzz, agent-task, and cost capabilities; these are not implemented and are omitted. Edges are limited to relationships shown in the excerpts or README.
Main entry points: cli.py CLI · connector.py Stdio connector · runner.py Suite runner · models.py Evaluation models
Components 11
Suite inputs
- YAML suites
suites - CLI
src/mcp_eval/cli.py - Suite loader
src/mcp_eval/suite.py
Test execution
- Suite runner
src/mcp_eval/runner.py - Stdio connector
src/mcp_eval/connector.py
Evaluation
- Evaluation models
src/mcp_eval/models.py - Check graders
src/mcp_eval/graders.py
Results
- Scorecard renderer
src/mcp_eval/report.py - JSON scorecard
Other components
- User
- MCP server
Connections 16
- User → CLI: runs commands
src/mcp_eval/cli.py - User → YAML suites: edits suites
README.md - YAML suites → Suite loader: loads YAML
README.md - CLI → Suite loader: loads suite
src/mcp_eval/cli.py - Suite loader → Evaluation models: creates cases
src/mcp_eval/suite.py - CLI → Suite runner: runs suite
src/mcp_eval/cli.py - Suite runner → Stdio connector: executes cases
src/mcp_eval/runner.py - Stdio connector → MCP server: sends tool calls
src/mcp_eval/connector.py - MCP server → Stdio connector: returns replies
src/mcp_eval/connector.py - Stdio connector → Evaluation models: records execution
src/mcp_eval/connector.py - Suite runner → Check graders: grades execution
src/mcp_eval/runner.py - Check graders → Evaluation models: creates verdict
src/mcp_eval/graders.py - Suite runner → Evaluation models: aggregates results
src/mcp_eval/runner.py - CLI → Scorecard renderer: renders scorecard
src/mcp_eval/cli.py - CLI → JSON scorecard: writes JSON
src/mcp_eval/cli.py - CLI → Stdio connector: discovers tools
src/mcp_eval/cli.py