# vii1nonly/mcp-eval-harness architecture

> Architecture diagram of the GitHub repository vii1nonly/mcp-eval-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vii1nonly/mcp-eval-harness
- Repository: https://github.com/vii1nonly/mcp-eval-harness

## Overview

mcp-eval is a reliability evaluation harness for MCP tool servers. A user selects a YAML suite and server command through the CLI; the suite loader creates test cases, the runner executes them through a stdio MCP connector, graders assign pass, fail, or inconclusive verdicts, and the CLI presents a scorecard. The CLI also supports tool discovery, while reports can be written as JSON. The README documents planned recovery, HTTP transport, security/fuzz, agent-task, and cost capabilities; these are not implemented and are omitted. Edges are limited to relationships shown in the excerpts or README.

Main entry points: [`src/mcp_eval/cli.py`](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/cli.py) (CLI), [`src/mcp_eval/connector.py`](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/connector.py) (Stdio connector), [`src/mcp_eval/runner.py`](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/runner.py) (Suite runner), [`src/mcp_eval/models.py`](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/models.py) (Evaluation models)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Suite inputs"]
  node_suite_data["YAML suites"]
  node_cli["CLI<br/>[cli.py]"]
  node_suite_loader["Suite loader<br/>[suite.py]"]
end

subgraph group_execution["Test execution"]
  node_runner["Suite runner<br/>[runner.py]"]
  node_connector["Stdio connector<br/>[connector.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_models["Evaluation models<br/>[models.py]"]
  node_graders["Check graders<br/>[graders.py]"]
end

subgraph group_outputs["Results"]
  node_report["Scorecard renderer<br/>[report.py]"]
  node_json_file["JSON scorecard"]
end

node_user(("User"))
node_mcp_server["MCP server"]

node_user -->|"runs commands"| node_cli
node_user -->|"edits suites"| node_suite_data
node_suite_data -->|"loads YAML"| node_suite_loader
node_cli -->|"loads suite"| node_suite_loader
node_suite_loader -->|"creates cases"| node_models
node_cli -->|"runs suite"| node_runner
node_runner -->|"executes cases"| node_connector
node_connector -->|"sends tool calls"| node_mcp_server
node_mcp_server -->|"returns replies"| node_connector
node_connector -->|"records execution"| node_models
node_runner -->|"grades execution"| node_graders
node_graders -->|"creates verdict"| node_models
node_runner -->|"aggregates results"| node_models
node_cli -->|"renders scorecard"| node_report
node_cli -->|"writes JSON"| node_json_file
node_cli -->|"discovers tools"| node_connector

click node_suite_data "https://github.com/vii1nonly/mcp-eval-harness/tree/main/suites"
click node_cli "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/cli.py"
click node_suite_loader "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/suite.py"
click node_models "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/models.py"
click node_runner "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/runner.py"
click node_connector "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/connector.py"
click node_graders "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/graders.py"
click node_report "https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_suite_data,node_cli,node_suite_loader,node_user toneBlue
class node_runner,node_connector toneAmber
class node_models,node_graders,node_mcp_server toneMint
class node_report,node_json_file toneRose
```

## Components

### Suite inputs

- [YAML suites](https://github.com/vii1nonly/mcp-eval-harness/tree/main/suites): `suites`
- [CLI](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/cli.py): `src/mcp_eval/cli.py`
- [Suite loader](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/suite.py): `src/mcp_eval/suite.py`

### Test execution

- [Suite runner](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/runner.py): `src/mcp_eval/runner.py`
- [Stdio connector](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/connector.py): `src/mcp_eval/connector.py`

### Evaluation

- [Evaluation models](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/models.py): `src/mcp_eval/models.py`
- [Check graders](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/graders.py): `src/mcp_eval/graders.py`

### Results

- [Scorecard renderer](https://github.com/vii1nonly/mcp-eval-harness/blob/main/src/mcp_eval/report.py): `src/mcp_eval/report.py`
- **JSON scorecard**

### Other components

- **User**
- **MCP server**

## Connections

- User → CLI: runs commands (evidence: `src/mcp_eval/cli.py`)
- User → YAML suites: edits suites (evidence: `README.md`)
- YAML suites → Suite loader: loads YAML (evidence: `README.md`)
- CLI → Suite loader: loads suite (evidence: `src/mcp_eval/cli.py`)
- Suite loader → Evaluation models: creates cases (evidence: `src/mcp_eval/suite.py`)
- CLI → Suite runner: runs suite (evidence: `src/mcp_eval/cli.py`)
- Suite runner → Stdio connector: executes cases (evidence: `src/mcp_eval/runner.py`)
- Stdio connector → MCP server: sends tool calls (evidence: `src/mcp_eval/connector.py`)
- MCP server → Stdio connector: returns replies (evidence: `src/mcp_eval/connector.py`)
- Stdio connector → Evaluation models: records execution (evidence: `src/mcp_eval/connector.py`)
- Suite runner → Check graders: grades execution (evidence: `src/mcp_eval/runner.py`)
- Check graders → Evaluation models: creates verdict (evidence: `src/mcp_eval/graders.py`)
- Suite runner → Evaluation models: aggregates results (evidence: `src/mcp_eval/runner.py`)
- CLI → Scorecard renderer: renders scorecard (evidence: `src/mcp_eval/cli.py`)
- CLI → JSON scorecard: writes JSON (evidence: `src/mcp_eval/cli.py`)
- CLI → Stdio connector: discovers tools (evidence: `src/mcp_eval/cli.py`)