How scaleapi/mcp-atlas works
MCP-Atlas is a benchmark system for measuring agents’ tool-use ability against real MCP servers. Its main workflow runs benchmark tasks through a TypeScript agent harness connected to a Docker sandbox, records conversations and responses, then scores answers with an LLM judge and can diagnose failures. The sandbox exposes tools from configured MCP servers; the harness alternates model completions and tool calls. The graph preserves documented evaluation and analysis stages. Some links are based on README workflow descriptions because those implementation excerpts were not provided.
Main entry points: agent-eval.ts Agent evaluation loop · run_eval.py Evaluation runner · score_claims.py Claim coverage scorer · single_model_diagnostic.py Failure diagnosis
Components 22
Evaluation workflow
- Benchmark tasks
- Evaluation runner
run_eval.py - Run results
Agent harness
- Harness API
services/agent-harness/src/index.ts - Agent evaluation loop
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts - Completion strategy
services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts - LiteLLM adapter
services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts - Sandbox MCP client
services/agent-harness/src/mcp-agent/helpers/mcp-client/sandbox-client.ts - Request and tool schemas
services/agent-harness/src/mcp-agent/schema.ts - Harness configuration
services/agent-harness/src/config.ts - Harness logging
services/agent-harness/src/logger.ts
MCP sandbox
- Sandbox API
services/agent-environment/src/agent_environment/main.py - Server client and config
services/agent-environment/src/agent_environment/mcp_client.py - MCP server set
services/agent-environment/src/agent_environment/mcp_server_template.json
Scoring and diagnostics
- Claim coverage scorer
services/scoring/score_claims.py - Scored results
- Failure diagnosis
services/diagnostics/single_model_diagnostic.py - Trajectory enrichment
services/diagnostics/extract_enriched_trajectory.py - Failure taxonomy
services/diagnostics/mcp_failure_taxonomy.py
Other components
- Benchmark operator
- Chat-completions model
- Scoring LLM
Connections 22
- Benchmark operator → Evaluation runner: starts run
README.md - Benchmark tasks → Evaluation runner: supplies tasks
README.md - Evaluation runner → Harness API: submits tasks
README.md - Harness API → Agent evaluation loop: routes requests
- Agent evaluation loop → Completion strategy: selects strategy
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts - Completion strategy → LiteLLM adapter: returns strategy
services/agent-harness/src/mcp-agent/agent-evals/completion-strategy.ts - Agent evaluation loop → Sandbox MCP client: lists and calls tools
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts - LiteLLM adapter → Chat-completions model: requests completion
services/agent-harness/src/mcp-agent/agent-evals/strategies/litellm-strategy.ts - Sandbox MCP client → Sandbox API: HTTP tool requests
README.md - Sandbox API → Server client and config: uses client
services/agent-environment/src/agent_environment/main.py - Server client and config → MCP server set: configures servers
services/agent-environment/src/agent_environment/mcp_client.py - Agent evaluation loop → Request and tool schemas: validates messages
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts - Evaluation runner → Run results: writes responses
README.md - Run results → Claim coverage scorer: provides model output
README.md - Benchmark tasks → Claim coverage scorer: provides ground truth
README.md - Claim coverage scorer → Scoring LLM: judges claims
README.md - Claim coverage scorer → Scored results: writes scores
README.md - Scored results → Failure diagnosis: provides failures
README.md - Failure diagnosis → Trajectory enrichment: enriches trajectory
services/diagnostics/single_model_diagnostic.py - Failure diagnosis → Failure taxonomy: classifies failures
services/diagnostics/single_model_diagnostic.py - Agent evaluation loop → Harness logging: records run events
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts - Agent evaluation loop → Harness configuration: reads settings
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts