How ysz/recursive-llm works
This repo is a Python library / SDK (not a deployed service) that implements Recursive Language Models (RLM) to let LLMs handle unbounded / very large contexts by keeping the full context in a Python variable and letting the model interactively explore it via a restricted Python REPL, including recursive sub-calls to cheaper models.
So the “system” you’re diagramming is primarily an in-process architecture:
• a user calls RLM.complete() / RLM.acomplete()
• the library orchestrates LLM calls (via LiteLLM) and safe local code execution (via RestrictedPython)
• the LLM iterates in a REPL loop until it emits FINAL(answer)
• Context is never sent wholesale to the provider
• The LLM sees instructions + query, and can “peek/search/slice” the in-memory context through executed code
• Recursive calls split context and solve subproblems
At runtime, the key loop is:
core.py initializes a REPL environment with
• context bound as a Python variable (the “unbounded context store”)
• helper functions, especially recursive_llm(...) that triggers nested RLM calls
prompts.py constructs the system prompt and “protocol”
• how to explore context via Python
• how to decide when to recurse
• requirement to finish with FINAL(answer)
core.py calls LiteLLM to reach an LLM provider
• root model: model=...
• optional recursive model: recursive_model=...
LLM returns a response containing either
• Python to execute (e.g., context[:1000], regex search, partitioning)
• or the terminal FINAL(...)
parser.py detects whether FINAL(...) is present.
• If FINAL: return to user
• Else: repl.py executes the code (restricted), captures output, feeds results back into the next LLM step
The loop continues until
• FINAL found, or
• max_iterations exceeded, or
• max_depth exceeded (for recursion), etc.
Recursion path:
• When model executes recursive_llm("subquery", context_slice), that creates a nested orchestrator call in core.py, usually using the cheaper recursive_model.
Async path:
Component details to explicitly annotate (RLM-specific, don’t omit)
• Context Handling: “Context stored in REPL variable; not appended to prompt”
• Safety Boundary: “RestrictedPython sandbox” around executing model code
• Protocol: “Model must output FINAL(...) to terminate”
• Limits / Guards: max_iterations, max_depth (and error path: “Max iterations exceeded”)
• Model Split: root model vs recursive model (cost optimization)
• Provider Abstraction: LiteLLM as the single integration surface (not direct OpenAI SDK usage)