How sakanaai/ab-mcts-arc2 works
Project Type and Purpose
• This is an open‐source research tool (CLI application + library) that:
• Runs Adaptive Branching Monte Carlo Tree Search (AB-MCTS) using multiple LLMs
• Solves ARC-AGI-2 benchmark tasks in parallel
• Processes and visualizes results
File Structure
• Top-level folders:
• src/ab_mcts_arc2 – core library (algorithm, LLM interfaces, prompts, tasks, utilities)
• experiments/arc2 – experiment runner & prompt definitions
• scripts – shell wrappers for launching experiments and evaluation
• Key files/configs:
• experiments/arc2/configs/config.yaml – LLM settings
• pyproject.toml & uv.lock – dependency management
• .gitmodules – TreeQuest submodule
• graphviz & parallel – external binaries for visualization & parallelism
README for Additional Insights
• Core workflow:
1. run_experiments.sh → launches experiments in parallel via experiments/arc2/run.py
2. experiment code uses TreeQuest AB-MCTS classes in src/ab_mcts_arc2
Main Components and Their Relationships
1. CLI Entry Points (scripts/run_experiments.sh, scripts/run_eval.sh)
2. Experiment Manager (experiments/arc2/run.py & prompt.py)
3. AB-MCTS Core Library (src/ab_mcts_arc2/):
• Algorithm classes (model_base.py, ab_mcts…)
• Data types & utilities (data_types.py, utils.py)
• Tasks & prompt modules (tasks/arc, prompts/…)
• LLM‐related modules: llm_interface, llm_builder, openai_api, claude_api, gemini_api
4. TreeQuest Submodule (external engine imported via .gitmodules)
5. External Services: OpenAI, Anthropic Claude, Google Gemini APIs
6. Data Store / File System:
• inputs: ARC-AGI-2 problem specs (bundled or downloaded)
• outputs: logs, solution files, processed results, plots
1. User invokes run_experiments.sh → Experiment Manager
2. Experiment Manager loads config.yaml → spawns AB-MCTS instances via TreeQuest
3. AB-MCTS pulls Tasks → generates Prompts → calls LLM Interfaces → receives completions
4. Algorithm expands search tree → writes candidate solutions to outputs/
5. Upon completion, user invokes run_eval.sh → proc_results.py reads outputs → generates metrics
c. Architectural Patterns & Design Principles:
• Separation of concerns: CLI vs. orchestration vs. core algorithm vs. evaluation
• Strategy/Plugin pattern for LLM providers (llm_interface, llm_builder)
• Config‐driven behavior (config.yaml controls LLM list & temperatures)
• Parallel processing (GNU parallel) for experiment scaling
• Modular prompts and tasks for extensibility to new problem domains
d. Key Technologies & Frameworks:
• Python packages managed by uv (pyproject.toml)
• TreeQuest algorithm (submodule)
• Graphviz (visualization)
• Shell tooling (bash, parallel)
• Blue – CLI & orchestration
• Green – core library
• Orange – external services
• Gray – file system / data stores
• “calls”, “reads config”, “generates prompts”, “sends completion request”, “writes output”, “processes results”, etc.
• UML-style annotations for key classes or modules if desired
Putting It All Together
• Middle layer: src/ab_mcts_arc2/** modules → TreeQuest engine → LLM API connectors → external LLMs
• Bottom left: local filesystem (inputs, outputs, logs)
• Bottom right: evaluation scripts → plots
• Data flow path end‐to‐end for a single experiment