How artificialray/cpu-kernel-baseline works
CPU-Kernel-Baseline evaluates agents that optimize AArch64 SIMD kernels. Its principal workflow selects benchmark definitions, provisions or reuses a target, starts an MCP kernel session, lets a harness compile and evaluate candidate code, and collects trajectories and results. The repository also supports a self-hosted litellm harness, kernel-definition and workload generation, and SIMD-loop benchmarks. Edges reflect README statements and source-index references or excerpts; several implementation details are unsampled, so uncertain connections are omitted.
Main entry points: bench_fleet.py Fleet driver · evaluator.py Litellm agent loop · launch_session.py Session launcher · base.py Kernel tool surface
Components 25
Run orchestration
- Fleet driver
test_scripts/bench_fleet.py - Harness adapters
test_scripts/harness_adapters.py - Session launcher
skills/launch/launch_session.py - Instance provisioning
provisioning/provision.py - Remote access
provisioning/remote.py
Agent interface
- Agent harness
- Litellm agent loop
eval/evaluator.py - LLM API dispatch
eval/llm_call.py - Model providers
eval/llm_providers.py - MCP client bridge
eval/mcp_client.py - MCP server
mcp_app/server.py - Session bootstrap
mcp_app/session.py - Kernel tool surface
mcp_app/agent_tools/base.py - ISA handling
mcp_app/agent_tools/isa.py
Kernel evaluation
- Compile and evaluate
mcp_app/agent_tools/ops.py - Kernel contracts
contracts.py
Workloads and kernels
- Model definitions
scripts/gen-definition/gen_model_definitions.py - Workload generation
scripts/gen-workload/gen_workload.py - Synthetic problem specs
dataset/problems/loop_012_particle_motion/problem.py - SIMD loop kernels
loops/loop_215.c - SIMD loop primitives
loops/common/loops.h
Results and studies
- Single-shot evaluation
eval/single_shot.py - Run logging
analysis/wandb_log_run.py - Baseline audits
analysis/audit_simd_baselines.py
Other components
- Benchmark operator
Connections 23
- Benchmark operator → Fleet driver: starts run
README.md - Fleet driver → Harness adapters: selects harness
test_scripts/bench_fleet.py - Fleet driver → Session launcher: prepares session
test_scripts/bench_fleet.py - Session launcher → Instance provisioning: provisions instance
skills/launch/launch_session.py - Session launcher → Remote access: uses remote access
skills/launch/launch_session.py - Harness adapters → Agent harness: invokes
README.md - Agent harness → MCP server: connects via MCP
README.md - Fleet driver → Litellm agent loop: drives own harness
README.md - Litellm agent loop → LLM API dispatch: requests completion
eval/evaluator.py - Litellm agent loop → Model providers: resolves provider
eval/evaluator.py - Litellm agent loop → MCP client bridge: uses MCP tools
eval/evaluator.py - MCP client bridge → MCP server: calls remotely
eval/mcp_client.py - MCP server → Session bootstrap: builds tools
mcp_app/server.py - Session bootstrap → Kernel tool surface: constructs session
mcp_app/session.py - Kernel tool surface → ISA handling: checks ISA
mcp_app/agent_tools/base.py - Kernel tool surface → Compile and evaluate: delegates operations
mcp_app/agent_tools/base.py - Compile and evaluate → Kernel contracts: uses policy
mcp_app/agent_tools/ops.py - Model definitions → Workload generation: generates definitions
- Synthetic problem specs → SIMD loop kernels: benchmarks kernels
- SIMD loop kernels → SIMD loop primitives: uses SIMD macros
loops/loop_215.c - Fleet driver → Run logging: logs runs
test_scripts/bench_fleet.py - Baseline audits → Compile and evaluate: audits baselines
analysis/audit_simd_baselines.py - Fleet driver → Single-shot evaluation: supports single-shot