How thunlp/schemareinforcementlearning works
Main Components
• Data & Schema Layer: Illustrate the raw schema data (downloaded from JSON Schema Store and GitHub) and the organizational structure (schemabench/data & train/data folders). Indicate that data includes both “custom” and “complex” schemas along with translation tests.
• Data Preprocessing & Parsing: Create a block for the SyntaxParser module. This is responsible for handling different file types (CSV, HTML, JSON, YAML, etc.) and cleaning/parsing the schemas.
• Benchmark & Evaluation Modules: Distinguish two main evaluation components:
– The bench module for running evaluation cases (including benchmarks such as GSM8K, MATH500, MMLU, etc.).
– The berkeley-function-call-leaderboard subproject that handles function-call evaluation using AST analysis, multi-turn evaluations, and model handlers (both for OSS and proprietary models).
• Training Pipeline: Represent the training modules used for fine-tuning models (SFT) and Schema Reinforcement Learning (SRL). Include the train directory (with SFT training scripts and data) and the included submodule (modified PRIME).
• Utility & Scripts Layer: Show the support provided by various bash scripts in the “scripts” folder (init_env.sh, train_sft.sh, train_srl.sh, test scripts) which orchestrate the overall workflow.
Relationships and Data Flow
• Data flows from the downloaded schema files (placed in the proper directories) into the SyntaxParser. The SyntaxParser transforms raw data according to supported syntaxes.
• Preprocessed data is then fed into the “bench” components which organize different testing benchmarks and evaluation routines.
• The training pipelines (SFT and SRL) use this data to fine-tune the language models. The “train” folder houses the training configurations and scripts which invoke sub-processes (e.g., via modified PRIME) for Schema RL.
• Trained models are then hooked up to the evaluation system, where CodeLinker (an external inference tool) is configured via a private configuration file and used in shell scripts (test_schemabench.sh) to run the evaluations.
• The berkeley-function-call-leaderboard module, which is more elaborate and modular, provides detailed performance evaluation with multiple metrics. This module is connected with both the training outputs (models) and the evaluation inputs (test data).
Architectural Patterns & Design Principles
• Modular design: Each top-level module (SyntaxParser, bench, berkeley-function-call-leaderboard, train, scripts) is clearly separated, allowing independent development and testing.
• Extensibility: The design supports multiple syntaxes (one parser per file type) and additional model handlers (OSS and proprietary) that can be plugged into the evaluation framework.
• Pipeline orchestration: The training and evaluation components are orchestrated using bash scripts, ensuring reproducible workflows.
• Integration of External Tools: The use of submodules (for the modified PRIME) and integration with CodeLinker for inference demonstrate both internal and external dependency flows.
Relevant Technologies & Frameworks
• Language: Python (with Python 3.11 specific environment setup).
• Scripting & Orchestration: Bash scripts are used to initialize environment, start training, and run evaluation.
• External Integrations: CodeLinker supports OpenAI compatible servers.
• Submodule: Modified version of PRIME for SRL is included as a Git submodule for model training.
• Data Formats: A wide range of schema formats (CSV, HTML, JSON, Markdown, etc.) are supported by the parser, highlighting a versatile parsing framework.