How bjterry/boardgamebench works
Project Type and Purpose
• This is a backend‐centric evaluation framework (CLI tool + library) for running automated games between LLMs, recording results, and computing Elo ratings.
• Purpose: orchestrate matches, interact with LLM providers, persist match state and ratings in a database, and export experiment results.
High‐Level Components
a. CLI / Entry Point
• bgbench/main.py
• Supporting scripts: init_postgres.py, generate_migration.py
b. Configuration & Environment
• .env, .env.example, config.py, bgbench/config_loader.py
c. Data Layer (Persistence)
• ORM models & migrations: bgbench/data/models.py, Alembic configs (alembic.ini, migrations/)
• Database initialization/migration scripts: bgbench/data/init_db.py, bgbench/data/migrate_db.py
• Supported backends: SQLite (default) or PostgreSQL
d. Domain Logic
• Game abstractions & implementations: bgbench/games/*.py (e.g., chess, nim, poker)
• Match management: bgbench/match/ (state manager, runner, view)
• Experiment orchestration: bgbench/experiment/ (arena, scheduler, rating, export)
• LLM integration: bgbench/llm/integration.py, bgbench/llm/player.py
e. Rating Engine
• Bayesian Elo: bgbench/experiment/rating.py
f. Export & Reporting
• bgbench/experiment/export.py
g. Utilities
• Serialization: bgbench/serialization.py
• Logging config: bgbench/logging_config.py
External Dependencies & Services
• LLM Providers: OpenAI, Anthropic, OpenRouter (via litellm)
• Database: SQLite or PostgreSQL
• CLI / OS environment
Component Relationships and Data Flow
1. User invokes CLI (bgbench/main.py) with parameters (e.g., --game nim).
2. Config Loader reads environment and YAML experiment config.
3. Experiment Scheduler selects pairs of LLM players for matches.
4. Match Runner:
• Instantiates Game subclass + two LLMPlayer instances.
• Loops: Get move prompt → call LLM integration → validate moves → update MatchState.
• Stores intermediate and final states via Data Layer.
5. Rating Module consumes match results and updates Elo ratings in DB.
6. Exporter writes experiment summary to disk (CSV/JSON).
7. Logging at each stage for audit/troubleshooting.
Architectural Patterns & Principles
• Modular “plugin” pattern for games: each game is a subclass of Game, discovered via Python imports.
• Layered architecture: CLI → Orchestration (experiment) → Domain (games, matches) → Data Layer → External Services.
• Async support in scheduler/runner for concurrency.
• Separation of concerns: LLM integration decoupled from game logic.
• Use of Alembic for database migrations (infrastructure as code).