How vii1nonly/mcp-assay works
mcp-assay is a command-line reliability evaluation harness for MCP tool servers. The principal workflow loads YAML suites into test cases, opens a stdio MCP session, executes each tool call, grades the observed result, aggregates a scorecard, and renders it to the console or JSON. Separate tools support server inspection; grading covers success, expected rejection, and output-schema checks, with inconclusive outcomes when answers are unobserved. The graph follows documented and excerpted calls; roadmap features are omitted because they are not implemented.
Main entry points: cli.py CLI commands · runner.py Suite runner · connector.py MCP connector · suite.py Suite loader
Components 15
CLI and reports
- CLI commands
src/mcp_assay/cli.py - Scorecard rendering
src/mcp_assay/report.py - JSON scorecard
src/mcp_assay/cli.py - Tool inspection
src/mcp_assay/cli.py
Suite definition
- YAML suites
suites - Suite loader
src/mcp_assay/suite.py - Test cases
src/mcp_assay/models.py
Execution
- Suite runner
src/mcp_assay/runner.py - MCP connector
src/mcp_assay/connector.py - Execution results
src/mcp_assay/models.py
Assessment
- Check graders
src/mcp_assay/graders.py - Graded results
src/mcp_assay/models.py - Scorecard aggregation
src/mcp_assay/models.py
Other components
- Operator
- MCP server
Connections 15
- Operator → CLI commands: runs command
README.md - CLI commands → Suite loader: loads suite
src/mcp_assay/cli.py - YAML suites → Suite loader: provides YAML
README.md - Suite loader → Test cases: creates cases
src/mcp_assay/suite.py - CLI commands → Suite runner: runs suite
src/mcp_assay/cli.py - Suite runner → MCP connector: executes calls
src/mcp_assay/runner.py - MCP connector → MCP server: calls tools
README.md - MCP connector → Execution results: records exchange
src/mcp_assay/connector.py - Suite runner → Check graders: grades result
src/mcp_assay/runner.py - Check graders → Graded results: creates verdict
src/mcp_assay/graders.py - Suite runner → Scorecard aggregation: aggregates results
src/mcp_assay/runner.py - CLI commands → Scorecard rendering: renders scorecard
src/mcp_assay/cli.py - CLI commands → JSON scorecard: writes JSON
src/mcp_assay/cli.py - CLI commands → Tool inspection: lists tools
src/mcp_assay/cli.py - Tool inspection → MCP server: inspects tools
src/mcp_assay/cli.py