How yueliu1999/guardreasoner works
Main Components Identification
• Data Storage & Organization:
• Benchmark and test data directories: “data/benchmark” and “data/test” harbor evaluation samples (for various safety tests such as SimpleSafetyTests, AegisSafetyTest, HarmBenchResponse, etc.) organized by model size (e.g., 1B, 3B, 8B).
• Training Pipeline:
• Scripts and tools in the “train” folder include shell scripts (e.g., “HS-DPO_1b.sh”, “R-SFT_8b.sh”), Python modules (e.g., “llm.py”, “chat_template.py”), and data preparation/synthesis scripts.
• This module handles both the reasoning SFT (Supervised Fine-Tuning) and hard sample DPO (Direct Preference Optimization) training methods.
• Evaluation Module:
• Standalone evaluation script “evaluate.py” which tests the safety performance across 13 benchmarks.
• Generation & Deployment:
• “generate.py” uses vLLM (or similar) to generate model responses.
• “deploy.py” enables the live or simulated deployment of the guard model.
• External Integration:
• Dependencies and inspirations include works from “LLaMA Factory”, “WildGuard”, and various LLaMA model releases, suggesting a reliance on pre-trained foundation models.
Relationships and Interactions
• Data Flow:
• Benchmark/test data serve as the input for both training and evaluation.
• The training pipeline ingests the GuardReasonerTrain dataset (127K samples) and uses it to fine-tune models.
• The output models (across scales: 1B, 3B, 8B) are then fed into the evaluation and generation/deployment modules.
• Pipeline Integration:
• Training scripts (in “train”) interact with data preparation and hard sample modules; once models are trained, they are evaluated using “evaluate.py”.
• “generate.py” interacts with the inference engine (likely leveraging vLLM) to produce model outputs, which are then optionally deployed via “deploy.py”.
• Architectural Flow:
• Clear sequential flow: Data Collection & Preparation -> Model Training (using both reasoning SFT and hard sample DPO) -> Model Evaluation -> Inference/Generation -> Deployment.
Architectural Patterns and Principles
• Modular Design:
• Each functional component (data, training, evaluation, generation, deployment) is separated into its own unit, making it easier to manage and evolve.
• Pipeline Architecture:
• The training and evaluation routines adhere to a pipeline model where data moves sequentially through predefined processing stages.
• Scalability:
• The multiple model sizes (1B, 3B, 8B) indicate that the system should be designed to handle different scales, possibly in parallel or using parameterized workflows.
• Extensibility:
• The organization of benchmarks and the variety of safety tests suggest that new tests or training strategies can be integrated with minimal disruption.
Technologies, Frameworks, or Libraries
• Python-based execution (e.g., “evaluate.py”, “generate.py”, “deploy.py”).
• Shell scripts for orchestrating training (e.g., “HS-DPO_x.sh”, “R-SFT_x.sh”).
• Likely integration with vLLM for generation/inference.
• Pre-trained models from LLaMA variants.
• JSON and JSONL for benchmarks and prediction outputs.
Detailed Elements
• Each module should include:
• Component name
• The main functionality (e.g., “train/prepare_data_rsft.py – preprocesses data for SFT”)
• Benchmark/test data locations and their role in evaluation.
• In the “Training Pipeline,” show the multiple training methods (R-SFT vs. HS-DPO) as sub-components.