How openai/following-instructions-human-feedback works
Project type and purpose
• This is a research‐oriented machine-learning pipeline for creating “InstructGPT” models by fine-tuning GPT-3 with human feedback.
• Purpose: collect labeler demonstrations, perform supervised learning, collect human rankings, perform reinforcement learning from human feedback (RLHF), and produce an aligned language model.
File structure
• Top level:
• README.md – describes the pipeline and links to paper
• model-card.md – metadata about the final InstructGPT models
• automatic-eval-samples/ – CSV files containing evaluation outputs on various NLP benchmarks
• No code folders present here, but from README we know there are modules for data collection, training (SL and RLHF), and evaluation.
README for additional insights
• Sections outline a two-stage training regime:
a. Supervised fine-tuning on “labeler demonstration” dataset
b. RLHF: train a reward model on human preference rankings, then optimize policy with PPO or similar
• External dependencies: GPT-3 API (for initial prompts and model calls), Google Docs for labeling instructions, human contractors.
a. Main components:
1. Data Collection
• Labeler-written prompts & demonstrations
• Prompts submitted via OpenAI API
• Human preference judgments (ranking dataset)
2. Supervised Fine-Tuning Module
• Input: demonstration dataset
• Output: “Supervised GPT-3” model
3. Reward Model Training
• Input: human rankings of model outputs
• Output: reward model
4. RLHF Training Module
• Inputs: supervised model, reward model
• Algorithm: PPO (or similar)
• Output: final InstructGPT model
5. Evaluation Module
• Runs final model on public NLP benchmarks
• Produces CSV sample outputs (automatic-eval-samples/)
6. Model Card Generation
• model-card.md
b. Relationships & interactions:
• Data Collection → feeds two parallel streams:
1. Supervised Fine-Tuning
2. Reward Model Training
• Supervised model + Reward model → RLHF Training → InstructGPT
• InstructGPT → Evaluation Module → sample CSV artifacts → model-card.md
c. Architectural patterns:
• Pipeline architecture (sequential, staged data flow)
• Modular separation: data collection, SL training, reward modeling, RL training, evaluation
d. Key technologies & dependencies:
• GPT-3 (OpenAI API)
• Human contractors & Google Docs labeling instructions
• ML frameworks (e.g., TensorFlow/PyTorch) for fine-tuning and RLHF
• CSV storage for evaluation artifacts
Detail
• Distinct dataset and model artifact as its own node
• External dependencies explicitly (human-in-the-loop, API)
• Sequential order: SL → Reward → RLHF → Evaluation → Documentation
• Avoid over-simplification; break down RLHF into Reward Model Training and Policy Optimization