How nkkko/ai-sandbox-benchmark works
Main Components
• Benchmark Runner & CLI/TUI: The entry points are the “benchmark.py” (for the interactive terminal UI) and “comparator.py” (CLI mode). They coordinate test execution.
• Providers Module: Contains the “providers” directory with multiple provider-specific scripts (e.g., daytona.py, codesandbox.py, e2b.py, modal.py, local.py) and additional helper files (like utils.py) for dealing with different ecosystem integrations.
• Tests and Metrics: The “tests” directory holds a range of performance tests (e.g., test_calculate_primes.py, test_fft_performance.py, etc.). Supporting scripts such as “metrics.py” help capture and analyze performance data.
• Configuration & Utility Files: Files like “config.yml” and the “.env.example” outline configuration for environment variables and sandbox settings. Additionally, there are migration scripts and specification documents.
Relationships and Interactions
• The Benchmark Runner (benchmark.py) reads configuration details (via config.yml and environment variables), loads tests from the “tests” directory, and triggers parallel execution across the different providers.
• Each Provider module encapsulates the interface to the sandbox environments. For example, “daytona.py” manages Daytona-related operations, while “codesandbox.py” (and its associated codesandbox-service.js) handles CodeSandbox.
• The Tests execute independent benchmark routines, and the “metrics.py” file gathers performance metrics (workspace creation, code execution, cleanup, etc.) which can then be aggregated and displayed via the comparator.
• There is a clear data flow: Configuration → Benchmark Runner/TUI → Parallel test execution across Providers → Metrics Collection → Results display (CLI/TUI).
Architectural Patterns & Design Principles
• Modularity: The clear separation of the “providers”, “tests”, and the central benchmarking execution logic indicates a modular design that allows adding new providers or tests.
• Parallelism: Tests are run in parallel across all selected providers, emphasizing concurrency.
• Configuration Driven: The use of files like “config.yml” to determine what parameters are passed to different sandboxes demonstrates flexibility.
• Reusability and Extensibility: With separate files for rules (inside the “.cursor/rules” directory) and a well-documented providers folder, the architecture is designed to easily incorporate additional benchmarks or provider integrations.