How naitchi/agent-smith works
Agent Smith solves MBPP coding exercises and SWE-bench repository issues through a Thought–Code–Observation loop. A benchmark CLI loads and validates a task, configures an LLM and sandbox, then the loop alternates model completions with sandbox execution until it has a solution; the CLI writes the result. The sandbox dynamically discovers MCP tools and proxies calls to the selected benchmark server. SWE-bench tools operate on a persistent Docker-backed repository, while MBPP tools provide the exercise-solving environment. The README documents both benchmark paths; unsampled implementation details are represented only where their roles and wiring are explicit.
Main entry points: __init__.py CLI and task helpers · provider.py LLM provider · agent_mbpp/__main__.py MBPP CLI · agent_swebench/__main__.py SWE-bench CLI
Components 24
Agent and benchmarks
- MBPP CLI
agent/agent_mbpp/__main__.py - SWE-bench CLI
agent/agent_swebench/__main__.py - CLI and task helpers
agent/__init__.py - MBPP task schema
schemas/mbpp_task_Input.py - SWE-bench task schema
schemas/swe_bench_task_input.py - Thought-code loop
src/agent_loop.py - Solution output schema
schemas/solution_output.py
Model access
- Provider registry
llm/registry.py - LLM provider
llm/provider.py - Key rotation
llm/rotator.py
Sandbox and MCP
- Sandbox runtime
src/sandbox/core.py - Persistent namespace
src/sandbox/namespace.py - Execution security
src/sandbox/security.py - MCP tool bridge
src/sandbox/mcp_bridge.py - Synchronous MCP client
src/mcp_sync_client.py - Async MCP client
src/mcp_client.py
Benchmark execution
- MBPP tool server
mcp_tools_mbpp.py - SWE-bench tool server
mcp_tools_swebench.py - Task repository container
src/docker_manager.py
Other components
- Benchmark operator
- LLM endpoint
- Docker engine
- Task repository
- Solution JSON file
Connections 28
- Benchmark operator → MBPP CLI: runs task
- Benchmark operator → SWE-bench CLI: runs issue
- MBPP CLI → CLI and task helpers: uses helpers
- SWE-bench CLI → CLI and task helpers: uses helpers
- CLI and task helpers → MBPP task schema: validates task
- CLI and task helpers → SWE-bench task schema: validates task
- MBPP CLI → Thought-code loop: runs loop
- SWE-bench CLI → Thought-code loop: runs loop
- CLI and task helpers → Provider registry: configures model
- Provider registry → LLM provider: creates provider
- LLM provider → Key rotation: rotates keys
- LLM provider → LLM endpoint: sends completion
- LLM endpoint → LLM provider: returns completion
- Thought-code loop → LLM provider: requests completion
- Thought-code loop → Sandbox runtime: executes code
- Sandbox runtime → Persistent namespace: runs code
- Sandbox runtime → Execution security: enforces limits
- Sandbox runtime → MCP tool bridge: bridges tool calls
- MCP tool bridge → Synchronous MCP client: dispatches requests
- Synchronous MCP client → Async MCP client: drives async calls
- Async MCP client → MBPP tool server: connects MCP
- Async MCP client → SWE-bench tool server: connects MCP
- SWE-bench tool server → Task repository container: runs repository tools
- Task repository container → Docker engine: manages container
- Docker engine → Task repository: hosts task checkout
- MBPP CLI → Solution output schema: builds result
- SWE-bench CLI → Solution output schema: builds result
- CLI and task helpers → Solution JSON file: writes JSON