# gfnorg/gfn-lm-tuning architecture

> Architecture diagram of the GitHub repository gfnorg/gfn-lm-tuning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gfnorg/gfn-lm-tuning
- Repository: https://github.com/gfnorg/gfn-lm-tuning

## Overview

##### High-Level Architecture Overview
The project consists of three main experimental components that should be represented as separate but related subsystems:
- Random Number Generation (RNG)
- Next Sentence Prediction
- Multi-purpose Infilling System (for stories, subjectivity classification, and arithmetic)

##### Key Components

a) Training Infrastructure
- Model Components:
- Base Language Models (GPT-2, GPT-J)
- GFlowNet Fine-tuning System
- PPO Training System (for RNG experiment)
- Supervised Fine-tuning (SFT) System

b) Data Processing Pipeline:
- Dataset Handlers (shown in data.py files)
- Data Generators
- Configuration Management (extensive YAML structure)

c) Evaluation Systems:
- Reward Computing Components
- Evaluation Scripts
- Results Analysis Tools

**┌─ Training Infrastructure ─┐**
- ┌─ Experiment Modules ─┐ │
- RNG  │ Next │ Infill│ │
- ┘ │
- ┘

##### Specific Implementation Details

For each experimental module, show:
- Configuration injection points
- Model checkpoint storage
- Evaluation feedback loops

##### Key Relationships to Highlight

- How the GFlowNet fine-tuning system interfaces with base models
- Relationship between configuration files and runtime behavior
- Demonstrate how the replay buffer system works with training
- Evaluation metrics flow

##### Technical Stack Elements

- PyTorch Lightning (evident from lightning_module.py)
- Hydra configuration system
- Base language models (GPT-2, GPT-J)
- Custom reward computation systems

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Training["Training Infrastructure"]
        LightningModule["Lightning Training System"]:::training
        SFT["SFT System"]:::training
        PPO["PPO Training"]:::training
        GFN["GFlowNet Training"]:::training
        Buffer["Replay Buffer"]:::training
    end

    subgraph Experiments["Experimental Modules"]
        RNG["RNG Experiment"]:::experiment
        NextSent["Next Sentence Prediction"]:::experiment
        Infill["Multi-purpose Infilling"]:::experiment
    end

    subgraph Data["Data Processing Pipeline"]
        InfillData["Infill Data Processor"]:::data
        NextSentData["Next Sentence Data Handler"]:::data
        RNGData["RNG Dataset Handler"]:::data
        DataGen["Data Generator"]:::data
    end

    subgraph Eval["Evaluation Systems"]
        ArithEval["Arithmetic Evaluation"]:::eval
        InfillEval["Infilling Evaluation"]:::eval
        RNGRewards["RNG Rewards"]:::eval
    end

    subgraph Config["Configuration Management"]
        ModelConfig["Model Configs"]:::config
        DatasetConfig["Dataset Configs"]:::config
        ExpConfig["Experiment Configs"]:::config
        DeviceConfig["Device Configs"]:::config
        BufferConfig["Buffer Config"]:::config
    end

    subgraph Utils["Utility Components"]
        RNGUtils["RNG Utils"]:::util
        NextSentUtils["Next Sentence Utils"]:::util
        InfillUtils["Infill Utils"]:::util
    end

    subgraph Results["Results Analysis"]
        RNGPlot["RNG Plotting"]:::results
        ResultsViz["Results Visualization"]:::results
    end

    %% Relationships
    ModelConfig --> Training
    DatasetConfig --> Data
    ExpConfig --> Experiments
    DeviceConfig --> Training
    BufferConfig --> Buffer

    Training --> Experiments
    Data --> Training
    Experiments --> Eval
    Utils --> Experiments
    Eval --> Results

    %% Click events
    click RNG "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/rng/"
    click NextSent "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/next_sentence/"
    click Infill "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/infill_subj_arithmetic/"
    click LightningModule "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/next_sentence/lightning_module.py"
    click SFT "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/sft.py"
    click PPO "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/notebooks/RNG_PPO.ipynb"
    click GFN "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/notebooks/RNG_GFN.ipynb"
    click InfillData "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/data.py"
    click NextSentData "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/next_sentence/lightning_data.py"
    click RNGData "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/rng_dataset.py"
    click DataGen "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/next_sentence/data/openwebtext/data_generator.py"
    click ArithEval "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/eval_arithmetic.py"
    click InfillEval "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/eval_infill.py"
    click RNGRewards "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/rng_rewards.py"
    click ModelConfig "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/infill_subj_arithmetic/configs/model/"
    click DatasetConfig "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/infill_subj_arithmetic/configs/dataset/"
    click ExpConfig "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/infill_subj_arithmetic/configs/experiment/"
    click DeviceConfig "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/next_sentence/configs/device/"
    click RNGUtils "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/rng_utils.py"
    click NextSentUtils "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/next_sentence/utils.py"
    click InfillUtils "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/utils.py"
    click RNGPlot "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/rng/rng_plot.py"
    click ResultsViz "https://github.com/GFNOrg/gfn-lm-tuning/tree/main/rng/notebooks/plots/"
    click Buffer "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/replay_buffer.py"
    click BufferConfig "https://github.com/GFNOrg/gfn-lm-tuning/blob/main/infill_subj_arithmetic/configs/rationale_buffer.yaml"

    %% Styling
    classDef training fill:#ffa500,stroke:#333,stroke-width:2px
    classDef experiment fill:#90EE90,stroke:#333,stroke-width:2px
    classDef data fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef eval fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef util fill:#FFE4B5,stroke:#333,stroke-width:2px
    classDef results fill:#98FB98,stroke:#333,stroke-width:2px
```
