How uiuc-kang-lab/revisql works
1. Repository purpose
- This repo packages the full ReViSQL pipeline for text-to-SQL: dataset curation, RLVR fine-tuning with execution-based reward, and inference-time reconciliation/majority voting.
- The primary execution path is: curated datasets + database schemas → training checkpoints → multi-candidate SQL generation → DB execution/evaluation → LLM-based reconciliation → final answered SQL.
2. Top-level architectural split
- data/ is the curated artifact layer: JSON/Parquet datasets plus schema SQL/metadata.
- data_scripts/ is the offline preprocessing layer that builds those artifacts.
- scripts/ contains the two operational entrypoints: training and inference.
- reconciliation/ is the post-generation decision layer used at inference time.
- tinker_cookbook/ is a bundled framework dependency that provides RL, supervised training, evaluation, tool-use, sandbox, and model-rendering infrastructure.
3. Data layer and schema boundary
- The repo tracks a large set of task datasets in data/, including BIRD-Verified splits, BIRD evaluation sets, Spider 2-SQLite and Spider 2-Snow evaluation sets, curriculum stages, and various ablation subsets.
- data/ddls/ holds per-database schema definitions used to execute SQL against the correct backend.
- For Spider 2-SQLite, the repo also includes spider2sqlite.json and spider2sqlite_tables.json, which act as schema metadata inputs for evaluation and execution.
4. Data preparation pipeline
- data_scripts/curate_final_data.py is the main curation step that produces bird-verified-train.parquet and val_test.parquet from the verified JSON sources.
- curate_RL_dataset.py, curate_curriculum_data.py, and the stage parquet files support earlier intermediate or curriculum-training variants.
- construct_db_schema.py and construct_spider2sqlite_db_schema.py generate executable schema DDLs, while generate_spider2sqlite_tables.py builds the Spider table metadata file.
- This preprocessing layer is offline and deterministic; it feeds the training and evaluation jobs but is not part of runtime inference logic.
5. Training subsystem
- scripts/train.py is the main RLVR fine-tuning entrypoint.
- It trains Qwen-family models with CISPO, using LoRA adapters, on bird-verified-train.parquet and validates on val_test.parquet.
- The training loop depends on local database execution via --db_path, so the reward signal is grounded in SQL correctness against real DB backends rather than text matching.
- Model checkpoints are written under models/, and the best checkpoint is selected by validation accuracy.
6. Inference subsystem
- scripts/infer.py loads a trained checkpoint and generates multiple candidate SQL queries per question using greedy and/or temperature-based sampling.
- The inference path is not a single-shot decode; it is a candidate-generation pipeline designed to support downstream grouping and reconciliation.
- Outputs are written to an output_path for later grading, analysis, and reconciliation.
- The same inference framework is used across BIRD, Spider 2-SQLite, and Spider 2-Snow by switching dataset and SQL engine configuration.
7. Execution-based evaluation boundary
- The core correctness signal is execution against the target database, not just string equality.
- This is why the repo needs both the question dataset and the matching per-database schema/DB files at runtime.
- For Spider 2-Snow, the architecture extends to a Snowflake execution backend; for the others, it runs against SQLite-style local databases.
- This execution layer is central to both training reward and inference evaluation.
8. Reconciliation subsystem
- reconciliation/ implements the paper’s inference-time “Decide” step: it inspects grouped candidate SQL queries and filters them by whether they satisfy the question constraints and any external knowledge hint.
- prompt.py contains the prompt templates, while reconciliation.py contains the decision logic.
- run_reconciliation.py and run_reconciliation_detailed.py are batch utilities for post-hoc decisioning and analysis.
- This stage uses a pre-RLVR base model as a judge, so it is an external LLM service call rather than a local deterministic rule engine.
9. Bundled framework: tinker_cookbook/
- This directory is a vendored ML/RL toolkit that supplies the reusable machinery for the project.
- Important sublayers include:
- rl/ for rollout, message-environment, metrics, and training loops
- supervised/ for SFT-style training
- preference/ and distillation/ for DPO and teacher/student workflows
- tool_use/, sandbox/, and recipes/ for environment-backed agent/tool execution
- renderers/ for model-specific chat/prompt formatting
- eval/ for evaluation utilities and inspectors
- For ReViSQL, the key architectural point is that the project builds on this framework rather than implementing training/evaluation primitives from scratch.
10. SQL-specific recipes and environments
- tinker_cookbook/recipes/sql_rl/ contains SQL-oriented environments, grading, prompts, correction logic, and Snowflake helpers.
- These modules are the bridge between generic RL infrastructure and the SQL execution domain.
- They encapsulate database interaction, SQL grading, and correction workflows so the training loop can remain model-agnostic.
- The presence of both sql_env.py and sql_correction_env.py indicates separate paths for raw SQL generation and correction-style interaction.
11. External services and operational dependencies
- Training/evaluation logging uses Weights & Biases via WANDB_API_KEY.
- Reconciliation uses external LLM providers through TOGETHER_API_KEY, with Groq as an optional alternative.
- Snowflake credentials are needed only for the Spider 2-Snow path.
- Local database artifacts are not stored in the repo; they must be mounted or downloaded and passed in via --db_path.
12. How to diagram the system
- A useful architecture diagram would show four main flows:
1. raw/corrected data and DDLs into data preparation,
2. prepared parquet datasets into RL training,
3. trained checkpoints into multi-sample inference,
4. candidate SQL groups into reconciliation and final selection.
- Surround these with the shared tinker_cookbook/ framework as the training/evaluation substrate, and annotate the external dependencies on local databases, W&B, and Together/Snowflake.
Main entry points: train.py Train · sql_rl SQL RL · infer.py Infer · reconciliation.py Reconcile
Components 21
Data layer
Curated datasets, schemas, and evaluation artifacts
- Curated data · Task datasets, curriculum stages, evaluation splits, and ablations
data - DB schemas · Per-database executable schema definitions for local SQL backends
data/ddls - Spider metadata · Spider 2-SQLite table and schema metadata inputs
data/spider2sqlite.json
Preparation
Offline curation and schema construction
- Final curation · Builds the main RLVR training and validation parquet sets
data_scripts/curate_final_data.py - RL curation · Prepares RL-oriented dataset variants
data_scripts/curate_RL_dataset.py - Curriculum prep · Constructs stage-wise curriculum datasets
data_scripts/curate_curriculum_data.py - Schema builders · Generate executable DDLs and Spider table metadata
data_scripts/construct_db_schema.py
Train & infer
Operational entrypoints and post-generation reconciliation
- Train · Main CISPO/LoRA fine-tuning job for Qwen-family models
scripts/train.py - Infer · Multi-sample SQL candidate generation for evaluation and reconciliation
scripts/infer.py - Judge prompt · Prompt templates for reconciliation decisions
reconciliation/prompt.py - Reconcile · Decision logic that filters grouped SQL candidates
reconciliation/reconciliation.py - Batch reconcile · Batch utilities for reconciliation runs and analysis
reconciliation/run_reconciliation.py
Framework
Vendored training, evaluation, and SQL RL substrate
- SQL RL · SQL-specific RL environments, grading, correction, and Snowflake helpers
tinker_cookbook/recipes/sql_rl - RL core · Rollouts, message environments, metrics, and training loops
tinker_cookbook/rl - Supervised · SFT data, evaluators, and training utilities
tinker_cookbook/supervised - Renderers · Model-specific chat and prompt formatting
tinker_cookbook/renderers - Tool use · Agent/tool execution abstractions
tinker_cookbook/tool_use - Eval utils · Evaluation and inspection helpers
tinker_cookbook/eval
Other components
- W&B · Training and evaluation logging
- LLM API · External judge model provider for reconciliation
- Snowflake · Execution backend for Spider 2-Snow
Connections 22
- Curated data → Final curation: source data
- DB schemas → Final curation: schemas
- Spider metadata → Schema builders: metadata
- Schema builders → DB schemas: generate
- RL curation → Curated data: variants
- Curriculum prep → Curated data: stages
- Final curation → Train: train/val parquet
- SQL RL → Train: SQL env
- RL core → Train: rollouts
- Supervised → Train: data/utilities
- Renderers → Train: chat format
- Tool use → SQL RL: tooling
- Train → W&B: logs
- Train → Infer: checkpoint
- Infer → SQL RL: execution
- Infer → Batch reconcile: candidate groups
- Batch reconcile → Reconcile: batch jobs
- Reconcile → Judge prompt: prompting
- Reconcile → LLM API: judge calls
- SQL RL → Snowflake: SnowSQL path
- Infer → Snowflake: Spider 2-Snow eval
- SQL RL → Eval utils: grading