How bird-bench/bird-rl works
1. What this repository is
BIRD-RL is a Python framework for training and evaluating SQL agents across a staged curriculum. The core boundary is between offline dataset construction, model training with VERL, and runtime inference/evaluation.
2. Primary training loop
The project is organized around four sequential training stages: reasoning SFT, multi-turn tool-use SFT, reasoning RL, and agentic RL. Each stage consumes a different data format but ultimately feeds the same VERL training stack through stage-specific scripts and reward functions.
3. Data preparation subsystem
bird_rl/data is the offline pipeline for turning raw task data and model outputs into training corpora. It generates prompts, calls external LLM APIs, parses responses, executes SQL to collect observations, builds trajectories, postprocesses them, and emits parquet-style datasets for SFT or RL.
4. Trajectory construction and supervision flow
The key data flow is: prompt generation → model/API response → response parsing → SQL execution/observation capture → trajectory assembly → validation/postprocessing → training dataset export. This is especially important for stage 2, where multi-turn agent traces become supervised examples.
5. Prompting layer
bird_rl/prompts centralizes prompt templates for the different task modes: reasoning-only, thought generation, multi-turn tool-calling, BIRD benchmark generation, and hybrid prompting. These templates define the model’s expected output structure, such as thought/solution blocks or tool-call formats.
6. Tooling and database boundary
bird_rl/tools and bird_rl/database implement the execution boundary between the language model and real SQL engines. The tools expose SQL execution and solution submission actions, while the database layer handles configuration, connections, pooling, and worker-safe session management.
7. Agentic RL runtime
Stage 4 depends on persistent, trajectory-scoped database sessions and cross-tool state. The verl_patch/tool_agent_loop_with_db_cleanup.py patch modifies VERL’s agent loop so tool use can retain and clean up database state correctly during interactive rollouts.
8. Reward computation
bird_rl/rewards contains reward functions for each task family and training mode: BIRD, Critic, and Hybrid, each with single-turn and agentic variants. These functions are the interface between generated trajectories and VERL’s RL optimization, converting task success and tool-feedback signals into training rewards.
9. Inference subsystem
bird_rl/inference mirrors the training pipeline for offline generation and evaluation. It includes vLLM batch inference, response parsing, SQL observation execution, and trajectory building, plus task-specific prompt/evaluation paths for BIRD and Critic inference.
10. Evaluation boundary
The evaluation/critic code is a separate evaluation harness for test-case-based SQL debugging metrics, while evaluation/bird points to the external BIRD benchmark workflow. This separation keeps benchmark scoring distinct from training data generation.
11. Scripts as orchestration
scripts/data, scripts/train, and scripts/infer are thin launch layers that compose the library modules into end-to-end workflows. They encode the operational pipeline order rather than new logic: data staging, training stage execution, and inference runs.
12. External dependencies and infrastructure
The project depends on VERL for RL training and vLLM for high-throughput inference. It also relies on external SQL databases and benchmark datasets such as BIRD-Critic-SQLite and BIRD mini-dev, which are the environment where tool execution and evaluation actually happen.
13. How to draw the architecture
A useful diagram would show three major planes: offline data/trajectory generation, training via VERL with reward and tool plugins, and inference/evaluation using the same prompt/parser/database abstractions. The database connection pool sits at the center of the tool-execution path, because both training rollouts and inference traces depend on it.
Main entry points: tools Tools · postprocess_trajectories.py Postprocess · rewards Rewards · call_api.py LLM calls
Components 31
Offline data
Curriculum data generation and trajectory assembly for all training stages.
- Data prep · Offline dataset construction pipeline from prompts and traces to trainable corpora.
bird_rl/data - Thought prompts · Generates reasoning-focused prompts for SFT and RL preparation.
bird_rl/data/generate_thought_prompts.py - Turn prompts · Generates multi-turn tool-use prompts and task scaffolds.
bird_rl/data/generate_turn_prompts.py - LLM calls · Calls external LLM APIs to populate trajectories and training examples.
bird_rl/data/call_api.py - Parse replies · Parses model outputs into structured turns and tool-call records.
bird_rl/data/parse_turn_responses.py - SQL obs · Executes SQL during data generation to capture observations for trajectories.
bird_rl/data/execute_sql_observations.py - Build trajectories · Assembles prompts, responses, and observations into trajectories.
bird_rl/data/build_trajectory.py - Postprocess · Validates and cleans trajectories before dataset export.
bird_rl/data/postprocess_trajectories.py
Training
Stage-specific training entrypoints, rewards, and VERL integration.
- Reasoning SFT · Stage 1 training entrypoint for reasoning-only supervised fine-tuning.
scripts/train/stage1_reasoning_sft.sh - Multi-turn SFT · Stage 2 training entrypoint for tool-use supervised fine-tuning.
scripts/train/stage2_multiturn_sft.sh - Reasoning RL · Stage 3 training entrypoint for reward-driven reasoning optimization.
scripts/train/stage3_reasoning_rl.sh - Agentic RL · Stage 4 training entrypoint for interactive tool-using RL.
scripts/train/stage4_agentic_rl.sh - VERL · External RL training stack used by all stages.
- Rewards · Task-specific reward functions for BIRD, Critic, and Hybrid modes.
bird_rl/rewards - BIRD reward · Rewards for BIRD task trajectories in single-turn and agentic variants.
bird_rl/rewards/bird_reward.py - Critic reward · Rewards for Critic task trajectories in single-turn and agentic variants.
bird_rl/rewards/critic_reward.py - Hybrid reward · Rewards for hybrid task trajectories in single-turn and agentic variants.
bird_rl/rewards/hybrid_reward.py
Inference & eval
Offline inference and benchmark evaluation paths that mirror training data flow.
- Inference · Batch inference and evaluation flow mirroring the training-side trajectory pipeline.
bird_rl/inference - vLLM batch · High-throughput batch generation for offline inference.
bird_rl/inference/vllm_infer.py - Parse outputs · Converts generated text into structured inference traces.
bird_rl/inference/parse_responses.py - SQL exec · Runs SQL during inference to obtain observations and feedback.
bird_rl/inference/execute_sql_observations.py - Build traces · Constructs inference trajectories for scoring and analysis.
bird_rl/inference/build_trajectory.py - BIRD eval · BIRD benchmark prompt generation and evaluation path.
bird_rl/inference/bird/evaluate.py - Critic eval · Critic benchmark evaluation path for debugging-style SQL tasks.
bird_rl/inference/critic/evaluate.py
Execution infra
Database and tool boundary used by both training rollouts and inference.
- Tools · Model-facing SQL execution and solution submission tools.
bird_rl/tools - DB pool · Worker-safe connection pooling and session management.
bird_rl/database/pool.py - DB config · Database configuration and connection setup.
bird_rl/database/config.py - Agent loop patch · VERL agent-loop patch for persistent tool state and database cleanup.
verl_patch/tool_agent_loop_with_db_cleanup.py - BIRD tools · Tool configuration for BIRD execution and submission.
configs/tools/bird_tools.yaml - Critic tools · Tool configuration for Critic execution and submission.
configs/tools/critic_tools.yaml - Hybrid tools · Tool configuration for hybrid execution and submission.
configs/tools/hybrid_tools.yaml
Connections 33
- Thought prompts → LLM calls: prompt · Reasoning prompts are sent to an external LLM API for generation.
- Turn prompts → LLM calls: prompt · Multi-turn prompts drive LLM generation for tool-use traces.
- LLM calls → Parse replies: responses · Generated responses are parsed into structured turn records.
- Parse replies → SQL obs: sql · Parsed turns trigger SQL execution to collect observations.
- SQL obs → Build trajectories: observations · SQL observations are assembled into trajectories.
- Build trajectories → Postprocess: clean · Trajectories are validated and normalized before export.
- Postprocess → Reasoning SFT: dataset · Processed corpora feed stage 1 supervised fine-tuning.
- Postprocess → Multi-turn SFT: dataset · Processed multi-turn traces feed stage 2 supervised fine-tuning.
- Postprocess → Reasoning RL: dataset · Processed trajectories feed stage 3 reward learning.
- Postprocess → Agentic RL: dataset · Processed agentic trajectories feed stage 4 RL.
- Reasoning SFT → VERL: train · Stage 1 is executed through the VERL training stack.
- Multi-turn SFT → VERL: train · Stage 2 is executed through the VERL training stack.
- Reasoning RL → VERL: train · Stage 3 uses VERL for RL optimization.
- Agentic RL → VERL: train · Stage 4 uses VERL for agentic RL rollouts.
- BIRD reward → Rewards: variant · BIRD-specific reward logic is part of the reward suite.
- Critic reward → Rewards: variant · Critic-specific reward logic is part of the reward suite.
- Hybrid reward → Rewards: variant · Hybrid reward logic is part of the reward suite.
- Rewards → VERL: signal · Reward functions provide optimization signals to VERL.
- Inference → vLLM batch: generate · Offline inference uses vLLM batch generation.
- vLLM batch → Parse outputs: outputs · Generated text is parsed into structured responses.
- Parse outputs → SQL exec: sql · Parsed inference traces may trigger SQL observation capture.
- SQL exec → Build traces: observations · Observations are assembled into inference trajectories.
- BIRD eval → Inference: path · BIRD evaluation reuses the inference pipeline.
- Critic eval → Inference: path · Critic evaluation reuses the inference pipeline.
- Tools → DB pool: sessions · Tool execution relies on pooled database sessions.
- DB config → DB pool: connect · Connection configuration feeds the pool.
- Agent loop patch → VERL: patch · VERL is patched to support persistent tool state during agentic rollouts.
- Tools → Agent loop patch: state · Tool state cleanup and persistence are coordinated through the patched loop.
- BIRD tools → Tools: config · BIRD-specific tool wiring configures the tool boundary.
- Critic tools → Tools: config · Critic-specific tool wiring configures the tool boundary.
- Hybrid tools → Tools: config · Hybrid-specific tool wiring configures the tool boundary.
- Tools → SQL obs: exec · Offline data generation depends on SQL tool execution.
- Tools → SQL exec: exec · Inference depends on SQL tool execution.