How artyompal/kaggle-abstract-reasoning works
Project Type and Purpose
• This is a hybrid C++/Python standalone application whose goal is to solve the Abstraction & Reasoning Challenge (ARC) puzzles via a domain-specific language (DSL) executed by a genetic algorithm (GA).
• The C++ core implements high-performance DSL operations (cellular automata and global transforms), exposed to Python via pybind11. The Python layer drives GA, test harnesses, and “bakes” a final Kaggle submission script.
File Structure
Top-level items:
• build.py – Python build script for compiling the C++/pybind11 extension
• gen_tables.sh – Generates auto-derived header (dsl_tables.h) for DSL primitives
• *.cpp/*.h – C++ DSL interpreter, helper utilities, JSON parser (json11.cpp/hpp), algorithms.h, static_array.h
• python-dsl-code/ – Pure-Python GA, DSL bindings (setup.py), bake_kernel.py, genetics.py
• crash_tests/ – JSON files for robustness testing
• tests/ – Shell scripts and solution data for testing correctness
• unit_tests.py – Python unit tests for core functionality
Extract Insights from README
• C++ port improved performance 50–100× over Python
• Uses pybind11 for Python extension
• GA in Python produces and scores candidate DSL programs
• bake_kernel.py generates a self-contained kernel.py for Kaggle
Components & Interactions
a. Main Components
1. Build System
– gen_tables.sh → dsl_tables.h
– build.py (invokes compiler + pybind11) → arc_dsl.so / arc_dsl.pyd
2. C++ Core Library (“arc_dsl”)
– DSL Parser/Interpreter (dsl.cpp/h)
– DSL primitives tables (dsl_tables.h)
– Cellular automata & global transforms (algorithms.h, static_array.h)
– JSON I/O via json11
3. Python GA Layer (python-dsl-code/)
– genetics.py: population, fitness evaluation (calls arc_dsl extension)
– setup.py: builds & installs Python package
– bake_kernel.py: emits final kernel.py
4. Test Harness
– crash_tests/ (raw task JSONs)
– tests/ (validation scripts and solution files)
– unit_tests.py: Python–level unit tests
5. External Service
– Kaggle Kernel: consumes kernel.py, produces submission.csv
b. Relationships & Data Flow
1. gen_tables.sh → (auto-generated) dsl_tables.h
2. build.py + C++ sources → arc_dsl Python extension
3. python-dsl-code/setup.py installs arc_dsl as Python package
4. genetics.py reads train/val tasks (JSON), invokes arc_dsl.evaluate(program, task)
5. arc_dsl runs DSL program on input image, returns output image
6. genetics.py computes fitness, applies selection, mutation, crossover
7. bake_kernel.py embeds best DSL program(s) into standalone kernel.py
8. kernel.py uploaded to Kaggle, executed there against hidden test set → submission.csv
c. Architectural Patterns & Principles
• Multi‐language modularization: C++ for performance-critical DSL engine; Python for orchestration, GA, and I/O.
• Code generation: DSL primitive tables auto-generated to simplify interpreter.
• Separation of concerns:
– Build scripts vs. runtime code
– Core evaluation engine vs. evolutionary search
– Testing harness vs. submission baking
d. Key Technologies & Libraries
• C++, pybind11 – Python binding for high-perf core
• json11.hpp – lightweight JSON parsing in C++
• Genetic Algorithm (custom in Python)
• Shell scripts for tests (filter.sh, restore.sh)
• Python packaging (setup.py)