How namin/holey works
Main Components
• Puzzle Solver (Main Entry Point): Represented by puzzle_solver.py – the driver that kicks off processing, reads inputs (e.g. puzzle names), and coordinates the solution process.
• Core Engine: Located in holey/core.py. This module includes key classes and logic for symbolic execution (tracers, symbolic representations, etc.). It’s the heart of the symbolic execution.
• Preprocessor: In holey/preprocessor.py contains AST node transformations and the SAT (satisfiability) driver. It processes the “holes” in the Python code before handing off to the solver.
• Backend for SMTLIB Generation: Found in holey/backend.py. This component creates SMTLIB (Satisfiability Modulo Theories Library) batch processes, interacting with solver engines (e.g., Z3 or cvc5).
• LLM Interface: In holey/llm.py, which manages LLM-based heuristics and controls involvement of multiple LLM providers (Ollama, Anthropic, Gemini, OpenAI). It plays a role in integrating natural language specifications.
• Tests: The tests directory (tests/test_core.py) indicates unit tests covering parts of the system.
• Benchmarks: The benchmarks directory (and its submodule of PythonProgrammingPuzzles) provides a source of problems used to evaluate the solver.
• Logging: Log files (in the log directory) show outputs, performance, and debugging data.
Interactions and Data Flow
• Start at puzzle_solver.py which orchestrates the execution by invoking the preprocessor, core, backend, and LLM modules.
• The preprocessor reads Python code with holes, performs AST transformations, and prepares the input for symbolic execution.
• The core engine executes symbolic analysis (branch exploration) and collaborates with the backend to generate SMTLIB programs.
• The backend is responsible for communicating with external SMT solvers (Z3, cvc5), handling batch processing.
• The LLM module is called (either in parallel or as a fallback) to assist in generating code completions or additional heuristics.
• Results and errors are logged in the log directory for review, while tests confirm that each component (especially the core engine and LLM integration as seen in tests/test_core.py) functions as expected.
• The benchmarking submodule provides sample puzzles that help validate performance and correctness.
Architectural Patterns and Principles
• Separation of Concerns: Each component (preprocessing, core analysis, backend processing, LLM integration) is modularized into its own Python file.
• Pipeline Flow: The process flows from accepting a puzzle, preprocessing/transforming code, applying symbolic execution through core, and then refining/augmenting through LLM interventions.
Relevant Technologies and Frameworks
• Python (with a note on required version 3.12 and pip/conda environment)
• SMT Solver Tools (Z3, cvc5)
• Various LLM providers (Ollama, Anthropic, Gemini, OpenAI)
• Unit testing framework (pytest could be implied by the tests/ directory)