How gfnorg/gfn-lm-tuning works
High-Level Architecture Overview
The project consists of three main experimental components that should be represented as separate but related subsystems:
• Random Number Generation (RNG)
• Next Sentence Prediction
• Multi-purpose Infilling System (for stories, subjectivity classification, and arithmetic)
Key Components
a) Training Infrastructure
• Model Components:
• Base Language Models (GPT-2, GPT-J)
• GFlowNet Fine-tuning System
• PPO Training System (for RNG experiment)
• Supervised Fine-tuning (SFT) System
b) Data Processing Pipeline:
• Dataset Handlers (shown in data.py files)
• Data Generators
• Configuration Management (extensive YAML structure)
c) Evaluation Systems:
• Reward Computing Components
• Evaluation Scripts
• Results Analysis Tools
┌─ Training Infrastructure ─┐
• ┌─ Experiment Modules ─┐ │
• RNG │ Next │ Infill│ │
• ┘ │
• ┘
Specific Implementation Details
For each experimental module, show:
• Configuration injection points
• Model checkpoint storage
• Evaluation feedback loops
Key Relationships to Highlight
• How the GFlowNet fine-tuning system interfaces with base models
• Relationship between configuration files and runtime behavior
• Demonstrate how the replay buffer system works with training
• Evaluation metrics flow
Technical Stack Elements
• PyTorch Lightning (evident from lightning_module.py)
• Hydra configuration system
• Base language models (GPT-2, GPT-J)
• Custom reward computation systems