How leonguertler/textarena works
Project type and purpose
• TextArena is a Python SDK/framework (packaged via pyproject.toml/setup.py) that provides an OpenAI-Gym–style interface for training, evaluating, and benchmarking models (LLMs or RL agents) in text-based games.
• It is not a web frontend/backend service, but a local library with optional online competition integration.
File structure
• Top-level:
• textarena/ – the core Python package
– api.py – factory methods (make, make_online, register_online_model)
– core.py – base Env definitions, registry logic
– registration.py – dynamic environment registration
– envs/… – individual game environments (each with its own env.py and optional renderer.py)
– agents/… – built-in agent implementations (e.g., OpenRouterAgent)
– wrappers/… – Gym-style wrappers for observations, actions, rendering
– utils/… – shared utilities (e.g., data loaders, jury logic)
• Examples: offline_play_example.py, online_play_example.py, samples/…
• Packaging/config: pyproject.toml, requirements.txt, .github/, setup.py, MANIFEST.in
README for architectural insights
• Core features: offline play loop, online play loop via remote HTTP API to textarena.ai
• Dependencies: external LLM APIs (OpenRouter, Claude, etc.), JSONL data for eval envs
Main components & interactions (
– textarena.api (user’s entrypoint: make(), make_online())
– textarena.core (Env base classes, step/reset semantics)
– textarena.envs.<GameID>
• env.py (game logic, state transitions)
• renderer.py (text‐based view)
• static data (e.g., prompts, word lists, JSONL data)
– ObservationWrappers (e.g., LLMObservationWrapper)
– ActionWrappers (e.g., formatting wrappers)
– RenderWrappers (e.g., simple console rendering)
– textarena.agents
• LLM agents (e.g., OpenRouterAgent making HTTP calls to external LLM APIs)
• Basic/random agents
– textarena.ai remote service (HTTP endpoints for model registration, online environments, leaderboard)
– On-disk JSONL/CSV assets under textarena/envs/.../data and textarena/utils/data
Interaction flow (offline):
User Script → textarena.api.make() → EnvFactory → Env Instance
Env.reset()
loop:
Env.get_observation() → ObservationWrapper → Agent(observation) → External LLM API → action
Env.step(action) → reward, done
Env.close()
Interaction flow (online):
User Script → textarena.api.make_online() → HTTP POST to textarena.ai → remote Env Instance
loop:
HTTP GET observation → Agent → HTTP POST action → remote Env.step → HTTP GET reward/done
HTTP POST close → remote logging/leaderboard
Architectural patterns and design principles
• Plugin/registry pattern: dynamic discovery of new envs via registration.py
• Decorator/wrapper pattern: modular, chainable wrappers for observations, actions, rendering
• Separation of concerns:
• Env logic (game rules)
• Wrappers (format conversion)
• Agents (decision making)
• API vs Core
Relevant technologies, frameworks, libraries
• Python 3.x, standard library
• Requests or httpx (for LLM calls and online HTTP API)
• OpenRouter/Anthropic/OpenAI client libraries
• JSONL for dataset storage
• PyPI packaging