How hkustdial/nl2sql360 works
NL2SQL360 is a testbed for importing NL2SQL benchmarks, evaluating model predictions, and querying fine-grained performance. The main workflow starts with a user invoking the CLI, parses command-specific configuration, imports dataset samples and SQL features, evaluates predictions against benchmark databases, stores results, and produces reports. Its evaluation surface includes Spider and BIRD metrics; reporting also supports query-feature scenarios and domain breakdowns. The sampled core and Spider evaluator clarify some wiring; several BIRD metric implementations and report internals are represented at their documented capability level.
Main entry points: core.py Evaluation Core · bird_eval BIRD Evaluation · util.py Command Handlers · cli.py CLI Commands
Components 18
User Workflow
- CLI Commands
src/nl2sql360/cli/cli.py - Command Handlers
src/nl2sql360/cli/util.py - Config Parsing
src/nl2sql360/arguments/parser.py - Evaluation Core
src/nl2sql360/core/core.py
Dataset And State
- Dataset Loader
src/nl2sql360/dataset/dataset.py - SQL Feature Parser
src/nl2sql360/parser/sql_parser.py - Result Models
src/nl2sql360/database/model.py - Evaluation Store
src/nl2sql360/core/core.py
Evaluation Metrics
- Spider EX And EM
src/nl2sql360/evaluator/spider_ex_em.py - Spider Test Suite
src/nl2sql360/evaluator/test_suite_sql_eval/evaluation.py - BIRD Evaluation
src/nl2sql360/evaluator/bird_eval - Soft F1
src/nl2sql360/evaluator/bird_eval/evaluation_f1.py - Valid Efficiency
- Reward Efficiency
Performance Analysis
- Scenario Filters
src/nl2sql360/filter/filter.py - Performance Reports
src/nl2sql360/core/core.py - History Deletion
src/nl2sql360/core/core.py
Other components
- Researcher
Connections 17
- Researcher → CLI Commands: invokes
- CLI Commands → Command Handlers: dispatches
- Command Handlers → Config Parsing: parses config
- Command Handlers → Evaluation Core: runs operation
- Evaluation Core → Dataset Loader: loads samples
- Evaluation Core → SQL Feature Parser: extracts features
- Evaluation Core → Result Models: creates tables
- Evaluation Core → Evaluation Store: reads and writes
- Evaluation Core → Spider EX And EM: invokes evaluation
- Spider EX And EM → Spider Test Suite: evaluates SQL
- Evaluation Core → BIRD Evaluation: runs metrics
- BIRD Evaluation → Soft F1: includes
- BIRD Evaluation → Valid Efficiency: includes
- BIRD Evaluation → Reward Efficiency: includes
- Evaluation Core → Scenario Filters: uses scenarios
- Evaluation Core → Performance Reports: queries results
- Evaluation Core → History Deletion: deletes history