How sakanaai/shinkaevolve works
1. Overall shape
- shinka is a Python framework for evolutionary program improvement driven by LLMs.
- The core loop is: start from an initial.<ext> program, generate candidate mutations with LLMs, evaluate them with a task-specific verifier, and feed the results back into selection, archive, and novelty mechanisms.
2. Primary entry points
- shinka_launch is the Hydra-based launcher for configured experiments.
- shinka_run is a task-directory CLI aimed at agent workflows and simpler ad hoc runs.
- ShinkaEvolveRunner is the main Python API for programmatic control.
- shinka_visualize serves the WebUI for live inspection of a running or completed experiment.
3. Task boundary
- Each task is external to the framework and supplies two key files: evaluate.py and the seed program (initial.py, initial.f90, initial.jl, etc.).
- evaluate.py is the verifier/scoring boundary; it executes the program under test, aggregates metrics, and may return text feedback.
- The framework treats the task program as the evolving artifact and the evaluator as the authoritative source of fitness.
4. Evolution core
- shinka/core contains the orchestration logic: proposal generation, async scheduling, timing, runtime slots, novelty checking, summarization, and evaluation wrapping.
- Candidate generation supports multiple patch styles such as diff-based, full-file, and cross-program edits.
- There is also a prompt-evolution path, where system prompts themselves can be evolved on a cadence.
5. LLM and mutation layer
- shinka/llm provides provider-agnostic access to model backends such as OpenAI, Anthropic, Gemini, DeepSeek, local OpenAI-compatible servers, and a headless CLI-backed mode.
- This layer also handles model resolution, pricing, retry-related constants, and model-selection prioritization.
- The rest of the system depends on this abstraction rather than talking to providers directly.
6. Edit application boundary
- shinka/edit is responsible for taking an LLM proposal and turning it into an executable candidate program.
- It supports applying patch diffs, replacing full files, and asynchronous edit execution.
- This is the boundary where LLM output becomes concrete code that can be evaluated.
7. Database, archive, and island model
- shinka/database manages persistent experiment state: archives of evaluated programs, parent/inspiration tracking, island sampling, and diversity-related selection.
- The architecture is island-based: multiple subpopulations evolve in parallel, with optional migration and dynamic spawning.
- Selection policies govern parent choice, archive replacement, and diversity/exploitation tradeoffs.
8. Embeddings and novelty
- shinka/embed provides code embeddings used for similarity checks and novelty gating.
- Novelty judgment can combine embedding similarity thresholds with optional LLM-based novelty review.
- This creates a second evaluation axis beyond raw task score: whether a candidate is sufficiently distinct or interesting.
9. Launch and execution environments
- shinka/launch abstracts where evaluation jobs run: locally, through SLURM, via conda activation, shell activation scripts, or Docker-on-SLURM.
- This is important because the evaluator is executed out-of-process, and the framework is designed to scale from laptop runs to cluster runs.
- shinka/cli and the Hydra configs under shinka/configs bind together task, database, cluster, and evolution presets into runnable experiments.
10. Asynchronous and throughput-focused mode
- The async runner path is a major subsystem: it decouples proposal generation, evaluation execution, and DB writes to improve throughput.
- Concurrency is intentionally split into separate budgets for evaluation jobs, proposal jobs, and database workers.
- The repo also includes oversubscription controls so proposal generation can be buffered when it is slower than evaluation.
11. Visualization and outputs
- shinka/webui contains the browser-based UI for inspecting lineages, trees, timing, costs, and performance trends.
- shinka/plots provides offline plotting utilities for costs, throughput, Pareto fronts, similarity, and lineage trees.
- The visual layer reads experiment artifacts and database state; it does not drive evolution itself.
12. Configuration model
- The system is heavily configuration-driven through Hydra presets and per-task YAMLs.
- Config domains are split by concern: task, database, evolution, cluster, and variant.
- This separation is what lets the same evolutionary engine run very different scientific tasks without changing core code.
13. Representative supported workflows
- The examples show the intended architecture: scientific or algorithmic tasks in Python, Fortran, or Julia; evaluation scripts; and a task-specific initial solution.
- The repository also includes “agent skills” wrappers that package common workflows like setup, conversion, run, and inspection for external coding agents.
- These skills are thin orchestration layers over the same core CLI and runner architecture.
14. External dependencies that matter architecturally
- LLM provider APIs, local OpenAI-compatible endpoints, embeddings services, and optional Headless CLI access are all first-class external integration points.
- SLURM and Docker are the main infrastructure dependencies for scaled execution.
- Markdown docs are generated with MkDocs, but that is documentation infrastructure rather than part of the runtime system.
Main entry points: core Core loop · database Experiment DB · async_runner.py Async runner · shinka_run Run CLI
Components 18
Runtime
Entry points and execution control
- Launch CLI · Hydra-based experiment launcher
shinka/cli/shinka_launch - Run CLI · Task-directory runner for ad hoc and agent workflows
shinka/cli/shinka_run - Visualize CLI · Serves the browser UI for experiment inspection
shinka/cli/shinka_visualize
Evolution engine
Core loop, mutation, evaluation, and prompt evolution
- Core loop · Coordinates proposal generation, scheduling, timing, and evaluation
shinka/core - Async runner · Separates proposal, evaluation, and DB work for parallel throughput
shinka/core/async_runner.py - Novelty gate · Checks distinctness using similarity and optional LLM review
shinka/core/novelty_judge.py - Prompt evo · Evolves system prompts on a cadence
shinka/core/prompt_evolver.py - Eval wrapper · Runs task evaluators and normalizes scoring/feedback
shinka/core/wrap_eval.py - Edit apply · Turns proposals into executable candidates via diff or full-file edits
shinka/edit
State and selection
Persistent archives, islands, parents, and novelty
- Experiment DB · Stores archives, parents, inspirations, islands, and prompts
shinka/database
Integrations
LLM providers, embeddings, and launch backends
- LLM layer · Provider-agnostic model access and routing
shinka/llm - Launch backends · Runs evaluators locally, under SLURM, or in containerized environments
shinka/launch - Embeddings · Generates embeddings for novelty and similarity checks
shinka/embed
Presentation
Web UI and offline plotting
Connections 21
- Launch CLI → Core loop: configures · Hydra launches the main evolution runner
- Run CLI → Async runner: starts · Task-directory CLI enters the async throughput path
- Visualize CLI → Web UI: serves · Starts the browser-based inspection UI
- Core loop → LLM layer: queries · Requests candidate proposals and prompt variants
- Core loop → Edit apply: mutates · Applies model output to produce candidate programs
- Core loop → Eval wrapper: evaluates · Wraps task-specific scoring and feedback
- Core loop → Experiment DB: persists · Writes archives, lineage, and selection state
- Core loop → Novelty gate: filters · Rejects or gates candidates by novelty
- Core loop → Prompt evo: adapts · Optionally evolves system prompts during the run
- Async runner → Launch backends: dispatches · Schedules out-of-process evaluator jobs
- Async runner → Experiment DB: streams · Decoupled worker writes and recovery handling
- Novelty gate → Embeddings: compares · Uses embeddings for similarity and diversity checks
- Experiment DB → Embeddings: indexes · Stores and samples with similarity-aware metadata
- Experiment DB → Core loop: feeds · Supplies parents, inspirations, and island sampling
- Launch backends → Eval wrapper: executes · Runs evaluators and task programs out-of-process
- Web UI → Experiment DB: reads · Visualizes lineage and state from experiment storage
- Plots → Experiment DB: analyzes · Generates offline summaries from stored runs
- Configs → Launch CLI: binds · Hydra presets assemble runnable experiments
- Configs → Run CLI: binds · Task configs and variants drive the task CLI
- Examples → Eval wrapper: defines · Each task supplies seed program and evaluate.py boundary
- Skills → Run CLI: wraps · Agent skills call into the same CLI and runner flow