# artyompal/kaggle-abstract-reasoning architecture

> Architecture diagram of the GitHub repository artyompal/kaggle-abstract-reasoning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/artyompal/kaggle-abstract-reasoning
- Repository: https://github.com/artyompal/kaggle-abstract-reasoning

## Overview

##### Project Type and Purpose
- This is a hybrid C++/Python standalone application whose goal is to solve the Abstraction & Reasoning Challenge (ARC) puzzles via a domain-specific language (DSL) executed by a genetic algorithm (GA).
- The C++ core implements high-performance DSL operations (cellular automata and global transforms), exposed to Python via pybind11. The Python layer drives GA, test harnesses, and “bakes” a final Kaggle submission script.

##### File Structure
Top-level items:
- build.py         – Python build script for compiling the C++/pybind11 extension
- gen_tables.sh    – Generates auto-derived header (dsl_tables.h) for DSL primitives
- *.cpp/*.h        – C++ DSL interpreter, helper utilities, JSON parser (json11.cpp/hpp), algorithms.h, static_array.h
- python-dsl-code/ – Pure-Python GA, DSL bindings (setup.py), bake_kernel.py, genetics.py
- crash_tests/     – JSON files for robustness testing
- tests/           – Shell scripts and solution data for testing correctness
- unit_tests.py    – Python unit tests for core functionality

##### Extract Insights from README
- C++ port improved performance 50–100× over Python
- Uses pybind11 for Python extension
- GA in Python produces and scores candidate DSL programs
- bake_kernel.py generates a self-contained kernel.py for Kaggle

##### Components & Interactions
a. Main Components
1. Build System
– gen_tables.sh → dsl_tables.h
– build.py (invokes compiler + pybind11) → arc_dsl.so / arc_dsl.pyd
2. C++ Core Library (“arc_dsl”)
– DSL Parser/Interpreter (dsl.cpp/h)
– DSL primitives tables (dsl_tables.h)
– Cellular automata & global transforms (algorithms.h, static_array.h)
– JSON I/O via json11
3. Python GA Layer (python-dsl-code/)
– genetics.py: population, fitness evaluation (calls arc_dsl extension)
– setup.py: builds & installs Python package
– bake_kernel.py: emits final kernel.py
4. Test Harness
– crash_tests/ (raw task JSONs)
– tests/ (validation scripts and solution files)
– unit_tests.py: Python–level unit tests
5. External Service
– Kaggle Kernel: consumes kernel.py, produces submission.csv

b. Relationships & Data Flow
1. gen_tables.sh → (auto-generated) dsl_tables.h
2. build.py + C++ sources → arc_dsl Python extension
3. python-dsl-code/setup.py installs arc_dsl as Python package
4. genetics.py reads train/val tasks (JSON), invokes arc_dsl.evaluate(program, task)
5. arc_dsl runs DSL program on input image, returns output image
6. genetics.py computes fitness, applies selection, mutation, crossover
7. bake_kernel.py embeds best DSL program(s) into standalone kernel.py
8. kernel.py uploaded to Kaggle, executed there against hidden test set → submission.csv

c. Architectural Patterns & Principles
- Multi‐language modularization: C++ for performance-critical DSL engine; Python for orchestration, GA, and I/O.
- Code generation: DSL primitive tables auto-generated to simplify interpreter.
- Separation of concerns:
– Build scripts vs. runtime code
– Core evaluation engine vs. evolutionary search
– Testing harness vs. submission baking

d. Key Technologies & Libraries
- C++, pybind11 – Python binding for high-perf core
- json11.hpp – lightweight JSON parsing in C++
- Genetic Algorithm (custom in Python)
- Shell scripts for tests (filter.sh, restore.sh)
- Python packaging (setup.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Build Pipeline
    subgraph "Build Pipeline"
        direction TB
        gen_tables["gen_tables.sh"]:::build
        dsl_tables["dsl_tables.h"]:::cppCore
        build_py["build.py"]:::build
        arc_ext["arc_dsl (C++/pybind11)"]:::cppCore
    end

    gen_tables -->|"generates"| dsl_tables
    build_py -->|"compiles"| arc_ext
    dsl_cpp["dsl.cpp"]:::cppCore
    dsl_h["dsl.h"]:::cppCore
    algorithms["algorithms.h"]:::cppCore
    static_array["static_array.h"]:::cppCore
    json11_cpp["json11.cpp"]:::cppCore
    json11_hpp["json11.hpp"]:::cppCore
    helpers["helpers.h"]:::cppCore

    dsl_cpp --> arc_ext
    dsl_h --> arc_ext
    dsl_tables --> arc_ext
    algorithms --> arc_ext
    static_array --> arc_ext
    json11_cpp --> arc_ext
    json11_hpp --> arc_ext
    helpers --> arc_ext

    %% Python Orchestration Zone
    subgraph "Python Orchestration" 
        direction TB
        setup_py["setup.py"]:::python
        genetics["genetics.py"]:::python
        bake["bake_kernel.py"]:::python
        kernel["kernel.py"]:::python
    end

    setup_py -->|"installs"| arc_ext
    genetics -->|"invokes evaluate() on"| arc_ext
    genetics -->|"feeds best program to"| bake
    bake -->|"emits"| kernel
    kernel -->|"uploaded to"| kaggle

    %% Test Harness
    subgraph "Test Harness"
        direction TB
        crash["crash_tests/"]:::data
        test_scripts["tests/ (filter.sh, restore.sh, solutions_train.txt, solutions_val.txt)"]:::data
        unit["unit_tests.py"]:::python
    end

    crash -.->|tasks for| genetics
    test_scripts -.->|validation for| arc_ext
    unit -->|tests| arc_ext

    %% External Service
    kaggle["Kaggle Kernel"]:::service

    %% Styles
    classDef build fill:#f9f,stroke:#333,stroke-width:1px;
    classDef cppCore fill:#cfc,stroke:#333,stroke-width:1px;
    classDef python fill:#ccf,stroke:#333,stroke-width:1px;
    classDef data fill:#eee,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef service fill:#ffc,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2;

    %% Click Events
    click gen_tables "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./gen_tables.sh"
    click dsl_tables "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./dsl_tables.h"
    click build_py "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./build.py"
    click dsl_cpp "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./dsl.cpp"
    click dsl_h "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./dsl.h"
    click algorithms "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./algorithms.h"
    click static_array "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./static_array.h"
    click json11_cpp "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./json11.cpp"
    click json11_hpp "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./json11.hpp"
    click helpers "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./helpers.h"
    click setup_py "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./setup.py"
    click genetics "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./python-dsl-code/genetics.py"
    click bake "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./python-dsl-code/bake_kernel.py"
    click crash "https://github.com/artyompal/kaggle-abstract-reasoning/tree/master/./crash_tests/"
    click test_scripts "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./tests/filter.sh"
    click test_scripts "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./tests/restore.sh"
    click test_scripts "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./tests/solutions_train.txt"
    click test_scripts "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./tests/solutions_val.txt"
    click unit "https://github.com/artyompal/kaggle-abstract-reasoning/blob/master/./unit_tests.py"
```
