# scaleapi/mcp-atlas architecture

> Architecture diagram of the GitHub repository scaleapi/mcp-atlas, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scaleapi/mcp-atlas
- Repository: https://github.com/scaleapi/mcp-atlas

## Overview

MCP-Atlas is a benchmark system for measuring agents’ tool-use ability against real MCP servers. Its main workflow runs benchmark tasks through a TypeScript agent harness connected to a Docker sandbox, records conversations and responses, then scores answers with an LLM judge and can diagnose failures. The sandbox exposes tools from configured MCP servers; the harness alternates model completions and tool calls. The graph preserves documented evaluation and analysis stages. Some links are based on README workflow descriptions because those implementation excerpts were not provided.

Main entry points: [`services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts) (Agent evaluation loop), [`run_eval.py`](https://github.com/scaleapi/mcp-atlas/blob/main/run_eval.py) (Evaluation runner), [`services/scoring/score_claims.py`](https://github.com/scaleapi/mcp-atlas/blob/main/services/scoring/score_claims.py) (Claim coverage scorer), [`services/diagnostics/single_model_diagnostic.py`](https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/single_model_diagnostic.py) (Failure diagnosis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation workflow"]
  node_tasks["Benchmark tasks"]
  node_runner["Evaluation runner<br/>[run_eval.py]"]
  node_responses["Run results"]
end

subgraph group_harness["Agent harness"]
  node_harness_api["Harness API<br/>[index.ts]"]
  node_agent_loop["Agent evaluation loop<br/>[agent-eval.ts]"]
  node_completion["Completion strategy"]
  node_litellm["LiteLLM adapter"]
  node_client["Sandbox MCP client<br/>[sandbox-client.ts]"]
  node_schemas["Request and tool schemas<br/>[schema.ts]"]
  node_config["Harness configuration<br/>[config.ts]"]
  node_logging["Harness logging<br/>[logger.ts]"]
end

subgraph group_sandbox["MCP sandbox"]
  node_environment["Sandbox API<br/>[main.py]"]
  node_mcp_client["Server client and config<br/>[mcp_client.py]"]
  node_servers["MCP server set"]
end

subgraph group_analysis["Scoring and diagnostics"]
  node_scorer["Claim coverage scorer<br/>[score_claims.py]"]
  node_scored["Scored results"]
  node_diagnostic["Failure diagnosis"]
  node_trajectory["Trajectory enrichment"]
  node_taxonomy["Failure taxonomy"]
end

node_researcher(("Benchmark operator"))
node_model["Chat-completions model"]
node_judge["Scoring LLM"]

node_researcher -->|"starts run"| node_runner
node_tasks -->|"supplies tasks"| node_runner
node_runner -->|"submits tasks"| node_harness_api
node_harness_api -->|"routes requests"| node_agent_loop
node_agent_loop -->|"selects strategy"| node_completion
node_completion -->|"returns strategy"| node_litellm
node_agent_loop -->|"lists and calls tools"| node_client
node_litellm -->|"requests completion"| node_model
node_client -->|"HTTP tool requests"| node_environment
node_environment -->|"uses client"| node_mcp_client
node_mcp_client -->|"configures servers"| node_servers
node_agent_loop -->|"validates messages"| node_schemas
node_runner -->|"writes responses"| node_responses
node_responses -->|"provides model output"| node_scorer
node_tasks -->|"provides ground truth"| node_scorer
node_scorer -->|"judges claims"| node_judge
node_scorer -->|"writes scores"| node_scored
node_scored -->|"provides failures"| node_diagnostic
node_diagnostic -->|"enriches trajectory"| node_trajectory
node_diagnostic -->|"classifies failures"| node_taxonomy
node_agent_loop -->|"records run events"| node_logging
node_agent_loop -->|"reads settings"| node_config

click node_runner "https://github.com/scaleapi/mcp-atlas/blob/main/run_eval.py"
click node_harness_api "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/index.ts"
click node_agent_loop "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts"
click node_completion "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts"
click node_litellm "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts"
click node_client "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/helpers/mcp-client/sandbox-client.ts"
click node_schemas "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/schema.ts"
click node_config "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/config.ts"
click node_logging "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/logger.ts"
click node_environment "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/main.py"
click node_mcp_client "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/mcp_client.py"
click node_servers "https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/mcp_server_template.json"
click node_scorer "https://github.com/scaleapi/mcp-atlas/blob/main/services/scoring/score_claims.py"
click node_diagnostic "https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/single_model_diagnostic.py"
click node_trajectory "https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/extract_enriched_trajectory.py"
click node_taxonomy "https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/mcp_failure_taxonomy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tasks,node_runner,node_responses toneBlue
class node_harness_api,node_agent_loop,node_completion,node_litellm,node_client,node_schemas,node_config,node_logging toneAmber
class node_environment,node_mcp_client,node_servers toneMint
class node_scorer,node_scored,node_diagnostic,node_trajectory,node_taxonomy toneRose
class node_researcher,node_model,node_judge toneIndigo
```

## Components

### Evaluation workflow

- **Benchmark tasks**
- [Evaluation runner](https://github.com/scaleapi/mcp-atlas/blob/main/run_eval.py): `run_eval.py`
- **Run results**

### Agent harness

- [Harness API](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/index.ts): `services/agent-harness/src/index.ts`
- [Agent evaluation loop](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts): `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`
- [Completion strategy](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts): `services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts`
- [LiteLLM adapter](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts): `services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts`
- [Sandbox MCP client](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/helpers/mcp-client/sandbox-client.ts): `services/agent-harness/src/mcp-agent/helpers/mcp-client/sandbox-client.ts`
- [Request and tool schemas](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/mcp-agent/schema.ts): `services/agent-harness/src/mcp-agent/schema.ts`
- [Harness configuration](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/config.ts): `services/agent-harness/src/config.ts`
- [Harness logging](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-harness/src/logger.ts): `services/agent-harness/src/logger.ts`

### MCP sandbox

- [Sandbox API](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/main.py): `services/agent-environment/src/agent_environment/main.py`
- [Server client and config](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/mcp_client.py): `services/agent-environment/src/agent_environment/mcp_client.py`
- [MCP server set](https://github.com/scaleapi/mcp-atlas/blob/main/services/agent-environment/src/agent_environment/mcp_server_template.json): `services/agent-environment/src/agent_environment/mcp_server_template.json`

### Scoring and diagnostics

- [Claim coverage scorer](https://github.com/scaleapi/mcp-atlas/blob/main/services/scoring/score_claims.py): `services/scoring/score_claims.py`
- **Scored results**
- [Failure diagnosis](https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/single_model_diagnostic.py): `services/diagnostics/single_model_diagnostic.py`
- [Trajectory enrichment](https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/extract_enriched_trajectory.py): `services/diagnostics/extract_enriched_trajectory.py`
- [Failure taxonomy](https://github.com/scaleapi/mcp-atlas/blob/main/services/diagnostics/mcp_failure_taxonomy.py): `services/diagnostics/mcp_failure_taxonomy.py`

### Other components

- **Benchmark operator**
- **Chat-completions model**
- **Scoring LLM**

## Connections

- Benchmark operator → Evaluation runner: starts run (evidence: `README.md`)
- Benchmark tasks → Evaluation runner: supplies tasks (evidence: `README.md`)
- Evaluation runner → Harness API: submits tasks (evidence: `README.md`)
- Harness API → Agent evaluation loop: routes requests
- Agent evaluation loop → Completion strategy: selects strategy (evidence: `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`)
- Completion strategy → LiteLLM adapter: returns strategy (evidence: `services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts`)
- Agent evaluation loop → Sandbox MCP client: lists and calls tools (evidence: `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`)
- LiteLLM adapter → Chat-completions model: requests completion (evidence: `services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts`)
- Sandbox MCP client → Sandbox API: HTTP tool requests (evidence: `README.md`)
- Sandbox API → Server client and config: uses client (evidence: `services/agent-environment/src/agent_environment/main.py`)
- Server client and config → MCP server set: configures servers (evidence: `services/agent-environment/src/agent_environment/mcp_client.py`)
- Agent evaluation loop → Request and tool schemas: validates messages (evidence: `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`)
- Evaluation runner → Run results: writes responses (evidence: `README.md`)
- Run results → Claim coverage scorer: provides model output (evidence: `README.md`)
- Benchmark tasks → Claim coverage scorer: provides ground truth (evidence: `README.md`)
- Claim coverage scorer → Scoring LLM: judges claims (evidence: `README.md`)
- Claim coverage scorer → Scored results: writes scores (evidence: `README.md`)
- Scored results → Failure diagnosis: provides failures (evidence: `README.md`)
- Failure diagnosis → Trajectory enrichment: enriches trajectory (evidence: `services/diagnostics/single_model_diagnostic.py`)
- Failure diagnosis → Failure taxonomy: classifies failures (evidence: `services/diagnostics/single_model_diagnostic.py`)
- Agent evaluation loop → Harness logging: records run events (evidence: `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`)
- Agent evaluation loop → Harness configuration: reads settings (evidence: `services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts`)
