# vii1nonly/mcp-assay architecture

> Architecture diagram of the GitHub repository vii1nonly/mcp-assay, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vii1nonly/mcp-assay
- Repository: https://github.com/vii1nonly/mcp-assay

## Overview

mcp-assay is a command-line reliability evaluation harness for MCP tool servers. The principal workflow loads YAML suites into test cases, opens a stdio MCP session, executes each tool call, grades the observed result, aggregates a scorecard, and renders it to the console or JSON. Separate tools support server inspection; grading covers success, expected rejection, and output-schema checks, with inconclusive outcomes when answers are unobserved. The graph follows documented and excerpted calls; roadmap features are omitted because they are not implemented.

Main entry points: [`src/mcp_assay/cli.py`](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py) (CLI commands), [`src/mcp_assay/runner.py`](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/runner.py) (Suite runner), [`src/mcp_assay/connector.py`](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/connector.py) (MCP connector), [`src/mcp_assay/suite.py`](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/suite.py) (Suite loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["CLI and reports"]
  node_cli["CLI commands<br/>[cli.py]"]
  node_report["Scorecard rendering<br/>[report.py]"]
  node_json_output["JSON scorecard<br/>[cli.py]"]
  node_tool_listing["Tool inspection<br/>[cli.py]"]
end

subgraph group_suite["Suite definition"]
  node_suite_data["YAML suites"]
  node_suite_loader["Suite loader<br/>[suite.py]"]
  node_test_case["Test cases<br/>[models.py]"]
end

subgraph group_execution["Execution"]
  node_runner["Suite runner<br/>[runner.py]"]
  node_connector["MCP connector<br/>[connector.py]"]
  node_execution_result["Execution results<br/>[models.py]"]
end

subgraph group_assessment["Assessment"]
  node_grader["Check graders<br/>[graders.py]"]
  node_graded_result["Graded results<br/>[models.py]"]
  node_scorecard["Scorecard aggregation<br/>[models.py]"]
end

node_user(("Operator"))
node_mcp_server(("MCP server"))

node_user -->|"runs command"| node_cli
node_cli -->|"loads suite"| node_suite_loader
node_suite_data -->|"provides YAML"| node_suite_loader
node_suite_loader -->|"creates cases"| node_test_case
node_cli -->|"runs suite"| node_runner
node_runner -->|"executes calls"| node_connector
node_connector -->|"calls tools"| node_mcp_server
node_connector -->|"records exchange"| node_execution_result
node_runner -->|"grades result"| node_grader
node_grader -->|"creates verdict"| node_graded_result
node_runner -->|"aggregates results"| node_scorecard
node_cli -->|"renders scorecard"| node_report
node_cli -->|"writes JSON"| node_json_output
node_cli -->|"lists tools"| node_tool_listing
node_tool_listing -->|"inspects tools"| node_mcp_server

click node_cli "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py"
click node_report "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/report.py"
click node_suite_data "https://github.com/vii1nonly/mcp-assay/tree/main/suites"
click node_suite_loader "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/suite.py"
click node_test_case "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py"
click node_runner "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/runner.py"
click node_connector "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/connector.py"
click node_execution_result "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py"
click node_grader "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/graders.py"
click node_graded_result "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py"
click node_scorecard "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py"
click node_json_output "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py"
click node_tool_listing "https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_report,node_json_output,node_tool_listing toneBlue
class node_suite_data,node_suite_loader,node_test_case toneAmber
class node_runner,node_connector,node_execution_result,node_mcp_server toneMint
class node_grader,node_graded_result,node_scorecard toneRose
class node_user toneIndigo
```

## Components

### CLI and reports

- [CLI commands](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py): `src/mcp_assay/cli.py`
- [Scorecard rendering](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/report.py): `src/mcp_assay/report.py`
- [JSON scorecard](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py): `src/mcp_assay/cli.py`
- [Tool inspection](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/cli.py): `src/mcp_assay/cli.py`

### Suite definition

- [YAML suites](https://github.com/vii1nonly/mcp-assay/tree/main/suites): `suites`
- [Suite loader](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/suite.py): `src/mcp_assay/suite.py`
- [Test cases](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py): `src/mcp_assay/models.py`

### Execution

- [Suite runner](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/runner.py): `src/mcp_assay/runner.py`
- [MCP connector](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/connector.py): `src/mcp_assay/connector.py`
- [Execution results](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py): `src/mcp_assay/models.py`

### Assessment

- [Check graders](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/graders.py): `src/mcp_assay/graders.py`
- [Graded results](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py): `src/mcp_assay/models.py`
- [Scorecard aggregation](https://github.com/vii1nonly/mcp-assay/blob/main/src/mcp_assay/models.py): `src/mcp_assay/models.py`

### Other components

- **Operator**
- **MCP server**

## Connections

- Operator → CLI commands: runs command (evidence: `README.md`)
- CLI commands → Suite loader: loads suite (evidence: `src/mcp_assay/cli.py`)
- YAML suites → Suite loader: provides YAML (evidence: `README.md`)
- Suite loader → Test cases: creates cases (evidence: `src/mcp_assay/suite.py`)
- CLI commands → Suite runner: runs suite (evidence: `src/mcp_assay/cli.py`)
- Suite runner → MCP connector: executes calls (evidence: `src/mcp_assay/runner.py`)
- MCP connector → MCP server: calls tools (evidence: `README.md`)
- MCP connector → Execution results: records exchange (evidence: `src/mcp_assay/connector.py`)
- Suite runner → Check graders: grades result (evidence: `src/mcp_assay/runner.py`)
- Check graders → Graded results: creates verdict (evidence: `src/mcp_assay/graders.py`)
- Suite runner → Scorecard aggregation: aggregates results (evidence: `src/mcp_assay/runner.py`)
- CLI commands → Scorecard rendering: renders scorecard (evidence: `src/mcp_assay/cli.py`)
- CLI commands → JSON scorecard: writes JSON (evidence: `src/mcp_assay/cli.py`)
- CLI commands → Tool inspection: lists tools (evidence: `src/mcp_assay/cli.py`)
- Tool inspection → MCP server: inspects tools (evidence: `src/mcp_assay/cli.py`)
