How sufe-aiflm-lab/fin-r1 works
Project Type and Purpose
• Project Type: A research/product release for a specialized financial reasoning LLM.
• Purpose: Leverage high-quality financial data and dual-stage training (supervised fine-tuning and reinforcement learning using GRPO) to enable robust financial decision-making and analysis in banking, securities, and other financial domains.
File Structure
• Configuration or code: Although there isn’t a rich codebase (the repository seems to be documentation-heavy), the technical details provided in the README serve as a blueprint for the data pipeline, training phases, and model serving instructions.
a. Main Components to Include:
• Data Construction & Distillation Module: This includes ingesting multiple diverse financial datasets (FinCorpus, FinQA, ConvFinQA, etc.) and processing them using a two-stage “答案+推理” (Answer + Reasoning) quality scoring mechanism.
• Data Filtering & Labeling: Separation of data into “good” (for SFT) and “bad” (for RL) based on rule-based matching and model scores.
• Reinforcement Learning (RL) Module: Using GRPO (Group Relative Policy Optimization), double-reward mechanisms (format and accuracy rewards), and model-based verification (via Qwen2.5-Max) to further refine the model.
• Evaluation & Benchmarking: Components showing how performance across different datasets (FinQA, ConvFinQA, etc.) is measured and compared.
• Deployment/Inference Service: Dedicated serving layer using vllm to load model weights and provide an API endpoint (as illustrated by the provided command-line and Python usage).
b. Relationships and Data Flows:
• Data flows into a “Data Distillation & Filtering” stage where quality gating is applied using two scoring rounds.
• From here, a bifurcation: one stream goes to SFT training, and the other (lower-quality or variant data) is used for RL training.
• The training modules are sequentially placed: First SFT to inject reasoning ability, then RL optimization to further enhance performance.
• Finally, the refined model is evaluated (comparing scores on benchmarks) and eventually deployed as a service that feeds user requests via API.
c. Architectural Patterns and Design Principles:
• Two-phase training pipeline (SFT then RL) is a critical architectural pattern.
• Emphasis on quality control via dual-stage data filtering.
• The “answer+reasoning” scheme highlights the importance of intermediate representations (chain-of-thought).
• The modular design allows a clear separation between data preprocessing, training/transformation, and deployment/inference.
d. Technologies, Frameworks, and Libraries:
• Training Algorithms: Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) for RL.
• Verification Mechanism: Qwen2.5-Max used in the model-based verifier.
• Deployment: vllm for serving the model with API endpoints.
• Clear Labels: Label every stage or component clearly (e.g., “Data Distillation Module”, “SFT Training”, “RL Optimization (GRPO)”, “Model-Based Verifier”, “Inference Server”).
• Annotations: Include brief text annotations indicating key processes such as “Dual-Stage Quality Scoring” in the data filtering module and “Double Reward Mechanism” in the RL training module.
• Integration Points: Clearly denote external integration points such as model download via HuggingFace and the serving layer command (vllm serve).