# openai/following-instructions-human-feedback architecture

> Architecture diagram of the GitHub repository openai/following-instructions-human-feedback, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/following-instructions-human-feedback
- Repository: https://github.com/openai/following-instructions-human-feedback

## Overview

##### Project type and purpose
- This is a research‐oriented machine-learning pipeline for creating “InstructGPT” models by fine-tuning GPT-3 with human feedback.
- Purpose: collect labeler demonstrations, perform supervised learning, collect human rankings, perform reinforcement learning from human feedback (RLHF), and produce an aligned language model.

##### File structure
- Top level:
- README.md – describes the pipeline and links to paper
- model-card.md – metadata about the final InstructGPT models
- automatic-eval-samples/ – CSV files containing evaluation outputs on various NLP benchmarks
- No code folders present here, but from README we know there are modules for data collection, training (SL and RLHF), and evaluation.

##### README for additional insights
- Sections outline a two-stage training regime:
a. Supervised fine-tuning on “labeler demonstration” dataset
b. RLHF: train a reward model on human preference rankings, then optimize policy with PPO or similar
- External dependencies: GPT-3 API (for initial prompts and model calls), Google Docs for labeling instructions, human contractors.

a. Main components:
1. Data Collection
- Labeler-written prompts & demonstrations
- Prompts submitted via OpenAI API
- Human preference judgments (ranking dataset)
2. Supervised Fine-Tuning Module
- Input: demonstration dataset
- Output: “Supervised GPT-3” model
3. Reward Model Training
- Input: human rankings of model outputs
- Output: reward model
4. RLHF Training Module
- Inputs: supervised model, reward model
- Algorithm: PPO (or similar)
- Output: final InstructGPT model
5. Evaluation Module
- Runs final model on public NLP benchmarks
- Produces CSV sample outputs (automatic-eval-samples/)
6. Model Card Generation
- model-card.md

b. Relationships & interactions:
- Data Collection → feeds two parallel streams:
1. Supervised Fine-Tuning
2. Reward Model Training
- Supervised model + Reward model → RLHF Training → InstructGPT
- InstructGPT → Evaluation Module → sample CSV artifacts → model-card.md

c. Architectural patterns:
- Pipeline architecture (sequential, staged data flow)
- Modular separation: data collection, SL training, reward modeling, RL training, evaluation

d. Key technologies & dependencies:
- GPT-3 (OpenAI API)
- Human contractors & Google Docs labeling instructions
- ML frameworks (e.g., TensorFlow/PyTorch) for fine-tuning and RLHF
- CSV storage for evaluation artifacts

##### Detail
- Distinct dataset and model artifact as its own node
- External dependencies explicitly (human-in-the-loop, API)
- Sequential order: SL → Reward → RLHF → Evaluation → Documentation
- Avoid over-simplification; break down RLHF into Reward Model Training and Policy Optimization

## Diagram (Mermaid)

```mermaid
graph TD
    %% Pipeline Documentation
    Doc["Pipeline Documentation"]:::artifact
    click Doc "https://github.com/openai/following-instructions-human-feedback/blob/main/README.md"

    %% External Services
    subgraph "External Services"
        direction TB
        Human["Human Labelers & Google Docs"]:::external
        API["OpenAI API"]:::external
    end

    %% Data Collection Stage
    DataCollection["Data Collection"]:::training
    Human -->|"labeling instructions"| DataCollection
    API -->|"prompt submissions"| DataCollection

    %% Data Repositories
    DemoCSV["Demonstrations CSV"]:::datastore
    RankingsCSV["Rankings CSV"]:::datastore
    DataCollection -->|"demonstrations CSV"| DemoCSV
    DataCollection -->|"rankings CSV"| RankingsCSV

    %% Supervised Fine-Tuning
    SFT["Supervised Fine-Tuning Module"]:::training
    DemoCSV -->|"input dataset"| SFT
    SFTModel["Supervised Model Checkpoint"]:::datastore
    SFT -->|"trained model checkpoint"| SFTModel

    %% Reward Model Training
    RMT["Reward Model Training"]:::training
    RankingsCSV -->|"input rankings CSV"| RMT
    RewardModel["Reward Model"]:::datastore
    RMT -->|"trained reward model"| RewardModel

    %% RLHF Training
    RLHF["RLHF Training Module"]:::training
    SFTModel -->|"pretrained policy"| RLHF
    RewardModel -->|"reward model"| RLHF
    RLHFModel["InstructGPT Model"]:::datastore
    RLHF -->|"final InstructGPT model"| RLHFModel

    %% Evaluation
    Evaluation["Evaluation Module"]:::artifact
    click Evaluation "https://github.com/openai/following-instructions-human-feedback/tree/main/automatic-eval-samples/"
    RLHFModel -->|"final model checkpoint"| Evaluation
    EvalCSV["Evaluation CSV Samples"]:::artifact
    Evaluation -->|"CSV sample outputs"| EvalCSV

    %% Model Card Generation
    ModelCard["Model Card Generation"]:::artifact
    click ModelCard "https://github.com/openai/following-instructions-human-feedback/blob/main/model-card.md"
    EvalCSV -->|"evaluation results"| ModelCard

    %% Legend
    classDef datastore fill:#D0E6FF,stroke:#004A9F,color:#004A9F;
    classDef training fill:#D5F5D5,stroke:#237A3E,color:#237A3E;
    classDef external fill:#FFE5CC,stroke:#B35C00,color:#B35C00;
    classDef artifact fill:#F0F0F0,stroke:#666666,color:#333333;
```
