# bird-bench/bird-rl architecture

> Architecture diagram of the GitHub repository bird-bench/bird-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bird-bench/bird-rl
- Repository: https://github.com/bird-bench/bird-rl

## Overview

1. **What this repository is**
   BIRD-RL is a Python framework for training and evaluating SQL agents across a staged curriculum. The core boundary is between offline dataset construction, model training with VERL, and runtime inference/evaluation.

2. **Primary training loop**
   The project is organized around four sequential training stages: reasoning SFT, multi-turn tool-use SFT, reasoning RL, and agentic RL. Each stage consumes a different data format but ultimately feeds the same VERL training stack through stage-specific scripts and reward functions.

3. **Data preparation subsystem**
   `bird_rl/data` is the offline pipeline for turning raw task data and model outputs into training corpora. It generates prompts, calls external LLM APIs, parses responses, executes SQL to collect observations, builds trajectories, postprocesses them, and emits parquet-style datasets for SFT or RL.

4. **Trajectory construction and supervision flow**
   The key data flow is: prompt generation → model/API response → response parsing → SQL execution/observation capture → trajectory assembly → validation/postprocessing → training dataset export. This is especially important for stage 2, where multi-turn agent traces become supervised examples.

5. **Prompting layer**
   `bird_rl/prompts` centralizes prompt templates for the different task modes: reasoning-only, thought generation, multi-turn tool-calling, BIRD benchmark generation, and hybrid prompting. These templates define the model’s expected output structure, such as thought/solution blocks or tool-call formats.

6. **Tooling and database boundary**
   `bird_rl/tools` and `bird_rl/database` implement the execution boundary between the language model and real SQL engines. The tools expose SQL execution and solution submission actions, while the database layer handles configuration, connections, pooling, and worker-safe session management.

7. **Agentic RL runtime**
   Stage 4 depends on persistent, trajectory-scoped database sessions and cross-tool state. The `verl_patch/tool_agent_loop_with_db_cleanup.py` patch modifies VERL’s agent loop so tool use can retain and clean up database state correctly during interactive rollouts.

8. **Reward computation**
   `bird_rl/rewards` contains reward functions for each task family and training mode: BIRD, Critic, and Hybrid, each with single-turn and agentic variants. These functions are the interface between generated trajectories and VERL’s RL optimization, converting task success and tool-feedback signals into training rewards.

9. **Inference subsystem**
   `bird_rl/inference` mirrors the training pipeline for offline generation and evaluation. It includes vLLM batch inference, response parsing, SQL observation execution, and trajectory building, plus task-specific prompt/evaluation paths for BIRD and Critic inference.

10. **Evaluation boundary**
   The `evaluation/critic` code is a separate evaluation harness for test-case-based SQL debugging metrics, while `evaluation/bird` points to the external BIRD benchmark workflow. This separation keeps benchmark scoring distinct from training data generation.

11. **Scripts as orchestration**
   `scripts/data`, `scripts/train`, and `scripts/infer` are thin launch layers that compose the library modules into end-to-end workflows. They encode the operational pipeline order rather than new logic: data staging, training stage execution, and inference runs.

12. **External dependencies and infrastructure**
   The project depends on VERL for RL training and vLLM for high-throughput inference. It also relies on external SQL databases and benchmark datasets such as BIRD-Critic-SQLite and BIRD mini-dev, which are the environment where tool execution and evaluation actually happen.

13. **How to draw the architecture**
   A useful diagram would show three major planes: offline data/trajectory generation, training via VERL with reward and tool plugins, and inference/evaluation using the same prompt/parser/database abstractions. The database connection pool sits at the center of the tool-execution path, because both training rollouts and inference traces depend on it.

Main entry points: [`bird_rl/tools`](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/tools) (Tools), [`bird_rl/data/postprocess_trajectories.py`](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/postprocess_trajectories.py) (Postprocess), [`bird_rl/rewards`](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/rewards) (Rewards), [`bird_rl/data/call_api.py`](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/call_api.py) (LLM calls)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_offline_data["Offline data"]
  node_data_prepare["Data prep<br/>offline pipeline"]
  node_thought_prompts["Thought prompts<br/>prompt gen"]
  node_turn_prompts["Turn prompts<br/>prompt gen"]
  node_call_api["LLM calls<br/>api client<br/>[call_api.py]"]
  node_parse_turn["Parse replies<br/>response parser"]
  node_exec_obs_data[("SQL obs<br/>sql exec")]
  node_build_traj_data["Build trajectories<br/>trajectory builder"]
  node_postprocess_data["Postprocess<br/>dataset transform"]
end

subgraph group_training["Training"]
  node_stage1_reasoning_sft["Reasoning SFT<br/>train stage"]
  node_stage2_multiturn_sft["Multi-turn SFT<br/>train stage"]
  node_stage3_reasoning_rl["Reasoning RL<br/>train stage"]
  node_stage4_agentic_rl["Agentic RL<br/>train stage"]
  node_train_verl["VERL<br/>rl trainer"]
  node_rewards{{"Rewards<br/>reward suite"}}
  node_bird_reward["BIRD reward<br/>reward fn<br/>[bird_reward.py]"]
  node_critic_reward["Critic reward<br/>reward fn<br/>[critic_reward.py]"]
  node_hybrid_reward["Hybrid reward<br/>reward fn<br/>[hybrid_reward.py]"]
end

subgraph group_runtime["Inference & eval"]
  node_inference_api["Inference<br/>offline runtime"]
  node_vllm_infer["vLLM batch<br/>model runtime<br/>[vllm_infer.py]"]
  node_parse_responses["Parse outputs<br/>response parser<br/>[parse_responses.py]"]
  node_exec_obs_infer[("SQL exec")]
  node_build_traj_infer["Build traces<br/>trajectory builder"]
  node_bird_eval["BIRD eval<br/>benchmark eval<br/>[evaluate.py]"]
  node_critic_eval["Critic eval<br/>benchmark eval<br/>[evaluate.py]"]
end

subgraph group_infra["Execution infra"]
  node_tools{{"Tools<br/>tool boundary"}}
  node_db_pool[("DB pool<br/>db infra<br/>[pool.py]")]
  node_db_conn["DB config<br/>[config.py]"]
  node_tool_agent_patch["Agent loop patch<br/>verl patch"]
  node_bird_tools_cfg["BIRD tools<br/>tool config<br/>[bird_tools.yaml]"]
  node_critic_tools_cfg["Critic tools<br/>tool config<br/>[critic_tools.yaml]"]
  node_hybrid_tools_cfg["Hybrid tools<br/>tool config<br/>[hybrid_tools.yaml]"]
end

node_thought_prompts -->|"prompt"| node_call_api
node_turn_prompts -->|"prompt"| node_call_api
node_call_api -->|"responses"| node_parse_turn
node_parse_turn -->|"sql"| node_exec_obs_data
node_exec_obs_data -->|"observations"| node_build_traj_data
node_build_traj_data -->|"clean"| node_postprocess_data
node_postprocess_data -->|"dataset"| node_stage1_reasoning_sft
node_postprocess_data -->|"dataset"| node_stage2_multiturn_sft
node_postprocess_data -->|"dataset"| node_stage3_reasoning_rl
node_postprocess_data -->|"dataset"| node_stage4_agentic_rl
node_stage1_reasoning_sft -->|"train"| node_train_verl
node_stage2_multiturn_sft -->|"train"| node_train_verl
node_stage3_reasoning_rl -->|"train"| node_train_verl
node_stage4_agentic_rl -->|"train"| node_train_verl
node_bird_reward -->|"variant"| node_rewards
node_critic_reward -->|"variant"| node_rewards
node_hybrid_reward -->|"variant"| node_rewards
node_rewards -->|"signal"| node_train_verl
node_inference_api -->|"generate"| node_vllm_infer
node_vllm_infer -->|"outputs"| node_parse_responses
node_parse_responses -->|"sql"| node_exec_obs_infer
node_exec_obs_infer -->|"observations"| node_build_traj_infer
node_bird_eval -->|"path"| node_inference_api
node_critic_eval -->|"path"| node_inference_api
node_tools -->|"sessions"| node_db_pool
node_db_conn -->|"connect"| node_db_pool
node_tool_agent_patch -->|"patch"| node_train_verl
node_tools -->|"state"| node_tool_agent_patch
node_bird_tools_cfg -->|"config"| node_tools
node_critic_tools_cfg -->|"config"| node_tools
node_hybrid_tools_cfg -->|"config"| node_tools
node_tools -->|"exec"| node_exec_obs_data
node_tools -->|"exec"| node_exec_obs_infer

click node_stage1_reasoning_sft "https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage1_reasoning_sft.sh"
click node_stage2_multiturn_sft "https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage2_multiturn_sft.sh"
click node_stage3_reasoning_rl "https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage3_reasoning_rl.sh"
click node_stage4_agentic_rl "https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage4_agentic_rl.sh"
click node_data_prepare "https://github.com/bird-bench/bird-rl/tree/main/bird_rl/data"
click node_thought_prompts "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/generate_thought_prompts.py"
click node_turn_prompts "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/generate_turn_prompts.py"
click node_call_api "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/call_api.py"
click node_parse_turn "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/parse_turn_responses.py"
click node_exec_obs_data "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/execute_sql_observations.py"
click node_build_traj_data "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/build_trajectory.py"
click node_postprocess_data "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/postprocess_trajectories.py"
click node_rewards "https://github.com/bird-bench/bird-rl/tree/main/bird_rl/rewards"
click node_bird_reward "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/bird_reward.py"
click node_critic_reward "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/critic_reward.py"
click node_hybrid_reward "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/hybrid_reward.py"
click node_inference_api "https://github.com/bird-bench/bird-rl/tree/main/bird_rl/inference"
click node_vllm_infer "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/vllm_infer.py"
click node_parse_responses "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/parse_responses.py"
click node_exec_obs_infer "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/execute_sql_observations.py"
click node_build_traj_infer "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/build_trajectory.py"
click node_bird_eval "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/bird/evaluate.py"
click node_critic_eval "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/critic/evaluate.py"
click node_tools "https://github.com/bird-bench/bird-rl/tree/main/bird_rl/tools"
click node_db_pool "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/database/pool.py"
click node_db_conn "https://github.com/bird-bench/bird-rl/blob/main/bird_rl/database/config.py"
click node_tool_agent_patch "https://github.com/bird-bench/bird-rl/blob/main/verl_patch/tool_agent_loop_with_db_cleanup.py"
click node_bird_tools_cfg "https://github.com/bird-bench/bird-rl/blob/main/configs/tools/bird_tools.yaml"
click node_critic_tools_cfg "https://github.com/bird-bench/bird-rl/blob/main/configs/tools/critic_tools.yaml"
click node_hybrid_tools_cfg "https://github.com/bird-bench/bird-rl/blob/main/configs/tools/hybrid_tools.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_prepare,node_thought_prompts,node_turn_prompts,node_call_api,node_parse_turn,node_exec_obs_data,node_build_traj_data,node_postprocess_data toneBlue
class node_stage1_reasoning_sft,node_stage2_multiturn_sft,node_stage3_reasoning_rl,node_stage4_agentic_rl,node_train_verl,node_rewards,node_bird_reward,node_critic_reward,node_hybrid_reward toneAmber
class node_inference_api,node_vllm_infer,node_parse_responses,node_exec_obs_infer,node_build_traj_infer,node_bird_eval,node_critic_eval toneMint
class node_tools,node_db_pool,node_db_conn,node_tool_agent_patch,node_bird_tools_cfg,node_critic_tools_cfg,node_hybrid_tools_cfg toneRose
```

## Components

### Offline data

Curriculum data generation and trajectory assembly for all training stages.

- [Data prep](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/data): offline pipeline · Offline dataset construction pipeline from prompts and traces to trainable corpora. · `bird_rl/data`
- [Thought prompts](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/generate_thought_prompts.py): prompt gen · Generates reasoning-focused prompts for SFT and RL preparation. · `bird_rl/data/generate_thought_prompts.py`
- [Turn prompts](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/generate_turn_prompts.py): prompt gen · Generates multi-turn tool-use prompts and task scaffolds. · `bird_rl/data/generate_turn_prompts.py`
- [LLM calls](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/call_api.py): api client · Calls external LLM APIs to populate trajectories and training examples. · `bird_rl/data/call_api.py`
- [Parse replies](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/parse_turn_responses.py): response parser · Parses model outputs into structured turns and tool-call records. · `bird_rl/data/parse_turn_responses.py`
- [SQL obs](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/execute_sql_observations.py): sql exec · Executes SQL during data generation to capture observations for trajectories. · `bird_rl/data/execute_sql_observations.py`
- [Build trajectories](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/build_trajectory.py): trajectory builder · Assembles prompts, responses, and observations into trajectories. · `bird_rl/data/build_trajectory.py`
- [Postprocess](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/data/postprocess_trajectories.py): dataset transform · Validates and cleans trajectories before dataset export. · `bird_rl/data/postprocess_trajectories.py`

### Training

Stage-specific training entrypoints, rewards, and VERL integration.

- [Reasoning SFT](https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage1_reasoning_sft.sh): train stage · Stage 1 training entrypoint for reasoning-only supervised fine-tuning. · `scripts/train/stage1_reasoning_sft.sh`
- [Multi-turn SFT](https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage2_multiturn_sft.sh): train stage · Stage 2 training entrypoint for tool-use supervised fine-tuning. · `scripts/train/stage2_multiturn_sft.sh`
- [Reasoning RL](https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage3_reasoning_rl.sh): train stage · Stage 3 training entrypoint for reward-driven reasoning optimization. · `scripts/train/stage3_reasoning_rl.sh`
- [Agentic RL](https://github.com/bird-bench/bird-rl/blob/main/scripts/train/stage4_agentic_rl.sh): train stage · Stage 4 training entrypoint for interactive tool-using RL. · `scripts/train/stage4_agentic_rl.sh`
- **VERL**: rl trainer · External RL training stack used by all stages.
- [Rewards](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/rewards): reward suite · Task-specific reward functions for BIRD, Critic, and Hybrid modes. · `bird_rl/rewards`
- [BIRD reward](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/bird_reward.py): reward fn · Rewards for BIRD task trajectories in single-turn and agentic variants. · `bird_rl/rewards/bird_reward.py`
- [Critic reward](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/critic_reward.py): reward fn · Rewards for Critic task trajectories in single-turn and agentic variants. · `bird_rl/rewards/critic_reward.py`
- [Hybrid reward](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/rewards/hybrid_reward.py): reward fn · Rewards for hybrid task trajectories in single-turn and agentic variants. · `bird_rl/rewards/hybrid_reward.py`

### Inference & eval

Offline inference and benchmark evaluation paths that mirror training data flow.

- [Inference](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/inference): offline runtime · Batch inference and evaluation flow mirroring the training-side trajectory pipeline. · `bird_rl/inference`
- [vLLM batch](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/vllm_infer.py): model runtime · High-throughput batch generation for offline inference. · `bird_rl/inference/vllm_infer.py`
- [Parse outputs](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/parse_responses.py): response parser · Converts generated text into structured inference traces. · `bird_rl/inference/parse_responses.py`
- [SQL exec](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/execute_sql_observations.py): sql exec · Runs SQL during inference to obtain observations and feedback. · `bird_rl/inference/execute_sql_observations.py`
- [Build traces](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/build_trajectory.py): trajectory builder · Constructs inference trajectories for scoring and analysis. · `bird_rl/inference/build_trajectory.py`
- [BIRD eval](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/bird/evaluate.py): benchmark eval · BIRD benchmark prompt generation and evaluation path. · `bird_rl/inference/bird/evaluate.py`
- [Critic eval](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/inference/critic/evaluate.py): benchmark eval · Critic benchmark evaluation path for debugging-style SQL tasks. · `bird_rl/inference/critic/evaluate.py`

### Execution infra

Database and tool boundary used by both training rollouts and inference.

- [Tools](https://github.com/bird-bench/bird-rl/tree/main/bird_rl/tools): tool boundary · Model-facing SQL execution and solution submission tools. · `bird_rl/tools`
- [DB pool](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/database/pool.py): db infra · Worker-safe connection pooling and session management. · `bird_rl/database/pool.py`
- [DB config](https://github.com/bird-bench/bird-rl/blob/main/bird_rl/database/config.py): db config · Database configuration and connection setup. · `bird_rl/database/config.py`
- [Agent loop patch](https://github.com/bird-bench/bird-rl/blob/main/verl_patch/tool_agent_loop_with_db_cleanup.py): verl patch · VERL agent-loop patch for persistent tool state and database cleanup. · `verl_patch/tool_agent_loop_with_db_cleanup.py`
- [BIRD tools](https://github.com/bird-bench/bird-rl/blob/main/configs/tools/bird_tools.yaml): tool config · Tool configuration for BIRD execution and submission. · `configs/tools/bird_tools.yaml`
- [Critic tools](https://github.com/bird-bench/bird-rl/blob/main/configs/tools/critic_tools.yaml): tool config · Tool configuration for Critic execution and submission. · `configs/tools/critic_tools.yaml`
- [Hybrid tools](https://github.com/bird-bench/bird-rl/blob/main/configs/tools/hybrid_tools.yaml): tool config · Tool configuration for hybrid execution and submission. · `configs/tools/hybrid_tools.yaml`

## Connections

- Thought prompts → LLM calls: prompt · Reasoning prompts are sent to an external LLM API for generation.
- Turn prompts → LLM calls: prompt · Multi-turn prompts drive LLM generation for tool-use traces.
- LLM calls → Parse replies: responses · Generated responses are parsed into structured turn records.
- Parse replies → SQL obs: sql · Parsed turns trigger SQL execution to collect observations.
- SQL obs → Build trajectories: observations · SQL observations are assembled into trajectories.
- Build trajectories → Postprocess: clean · Trajectories are validated and normalized before export.
- Postprocess → Reasoning SFT: dataset · Processed corpora feed stage 1 supervised fine-tuning.
- Postprocess → Multi-turn SFT: dataset · Processed multi-turn traces feed stage 2 supervised fine-tuning.
- Postprocess → Reasoning RL: dataset · Processed trajectories feed stage 3 reward learning.
- Postprocess → Agentic RL: dataset · Processed agentic trajectories feed stage 4 RL.
- Reasoning SFT → VERL: train · Stage 1 is executed through the VERL training stack.
- Multi-turn SFT → VERL: train · Stage 2 is executed through the VERL training stack.
- Reasoning RL → VERL: train · Stage 3 uses VERL for RL optimization.
- Agentic RL → VERL: train · Stage 4 uses VERL for agentic RL rollouts.
- BIRD reward → Rewards: variant · BIRD-specific reward logic is part of the reward suite.
- Critic reward → Rewards: variant · Critic-specific reward logic is part of the reward suite.
- Hybrid reward → Rewards: variant · Hybrid reward logic is part of the reward suite.
- Rewards → VERL: signal · Reward functions provide optimization signals to VERL.
- Inference → vLLM batch: generate · Offline inference uses vLLM batch generation.
- vLLM batch → Parse outputs: outputs · Generated text is parsed into structured responses.
- Parse outputs → SQL exec: sql · Parsed inference traces may trigger SQL observation capture.
- SQL exec → Build traces: observations · Observations are assembled into inference trajectories.
- BIRD eval → Inference: path · BIRD evaluation reuses the inference pipeline.
- Critic eval → Inference: path · Critic evaluation reuses the inference pipeline.
- Tools → DB pool: sessions · Tool execution relies on pooled database sessions.
- DB config → DB pool: connect · Connection configuration feeds the pool.
- Agent loop patch → VERL: patch · VERL is patched to support persistent tool state during agentic rollouts.
- Tools → Agent loop patch: state · Tool state cleanup and persistence are coordinated through the patched loop.
- BIRD tools → Tools: config · BIRD-specific tool wiring configures the tool boundary.
- Critic tools → Tools: config · Critic-specific tool wiring configures the tool boundary.
- Hybrid tools → Tools: config · Hybrid-specific tool wiring configures the tool boundary.
- Tools → SQL obs: exec · Offline data generation depends on SQL tool execution.
- Tools → SQL exec: exec · Inference depends on SQL tool execution.
