How ethereum/execution-spec-tests works
Project Type and Purpose
• This is a Python‐based test-fixture generation framework (an open-source tool) for Ethereum execution clients.
• It collects Python test cases, invokes external “t8n” executables (EVM transition tools like evm t8n, evmone-t8n, etc.), and outputs JSON fixtures that clients or the Ethereum Hive integration framework can consume.
High-Level Components
a. Command-Line Interface (CLI)
• src/cli: Implements fill, make, check_fixtures, order_fixtures, and other entry-points.
• Uses uv (external tool) to manage Python virtual environments.
b. Core Engine Modules
• src/ethereum_test_base_types, _fixtures, _execution, _forks, _rpc, _specs, _tools, _types, _vm: Organize functionality to parse tests, generate intermediate representations, drive the t8n tool invocation, and assemble the JSON output.
c. Pytest Plugins
• src/pytest_plugins: Provide custom pytest commands (fill, consume, execute, filler, forks, etc.) that tie test discovery and execution to fixture generation.
d. Configuration & Orchestration
• src/config and .github/configs: Define external tool versions (EIPs), YAML templates, environment variables, and CI workflows.
e. External Dependencies & Services
• Transition tools (e.g., evm t8n, evmone-t8n, ethereum-spec-evm, solc) invoked by the framework.
• GitHub Actions workflows for CI, CD, and documentation deployment.
f. Output Consumers
• JSON fixtures under fixtures/ are consumed by client test frameworks (ethereum/hive) or other EVM clients.
Relationships & Data Flows
• The user invokes a CLI command (uv run fill …).
• CLI parses arguments and loads configuration (src/config).
• Pytest plugin hooks into pytest to discover Python test cases (src/cli and src/ethereum_test_* modules).
• For each test, the framework calls the appropriate external t8n executable with test data.
• The t8n tool returns execution results; the framework then formats them into JSON fixtures.
• Fixtures are written to disk (fixtures/...).
• Downstream, those JSON files flow into client test harnesses or the Hive framework.
Architectural Patterns & Principles
• Modular packaging: Each major feature has its own Python namespace (ethereum_test_*).
• Plugin architecture: Custom pytest plugins extend pytest for test discovery and fixture generation.
• External tool abstraction: Adapters in src/ethereum_clis abstract over different t8n implementations.
• Configuration-as-code: YAML templates and Python config modules drive versions and environment setup.
• Separation of concerns: CLI layer, core engine, plugins, adapters, and configuration are clearly separated.