# eleutherai/gpt-neox architecture

> Architecture diagram of the GitHub repository eleutherai/gpt-neox, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eleutherai/gpt-neox
- Repository: https://github.com/eleutherai/gpt-neox

## Overview

##### High-Level Components

a) Core Components (Main Layer):
- Training Pipeline
- Evaluation System
- Generation System
- Data Processing Pipeline
- Model Architecture
- Distributed Training Infrastructure

b) Supporting Components (Secondary Layer):
- Monitoring Systems (WandB, TensorBoard, Comet)
- Checkpointing System
- Profiling Tools
- Configuration Management

##### Detailed Component Breakdown

a) Training Pipeline:
- Subcomponents:
- Megatron Core (model parallelism)
- DeepSpeed Integration (optimization & parallelization)
- Data Loading System
- Optimizer Management
- Learning Rate Scheduling

b) Model Architecture:
- Parallel branches for different architectures:
- GPT-NeoX Architecture
- LLaMA Architecture
- Mistral Architecture
- RWKV Architecture
- Mamba Architecture
- MoE (Mixture of Experts)

c) Distributed Training Infrastructure:
- Multiple deployment options:
- Multi-Node Training
- Different Launchers (Slurm, MPI, pdsh)
- GPU Communication Paths
- Pipeline Parallelism
- Tensor Parallelism
- Data Parallelism

- Training data flow
- Gradient flow
- Parameter synchronization
- Monitoring data
- Checkpoint data

##### Integration Points

- External system connections:
- Hugging Face Integration
- Monitoring Platforms
- Storage Systems (local/S3)
- Evaluation Harness

##### Specific Implementation Details

- Technical annotations for:
- Flash Attention Support
- Transformer Engine Integration
- Fused CUDA Kernels
- AMD GPU Support

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Core["Core Components"]
        TP["Training Pipeline"]
        ES["Evaluation System"]
        GS["Generation System"]
        DPP["Data Processing Pipeline"]
        MA["Model Architecture"]
        DTI["Distributed Training"]
    end

    subgraph Support["Supporting Components"]
        MS["Monitoring Systems"]
        CS["Checkpointing System"]
        PT["Profiling Tools"]
        CM["Configuration Management"]
    end

    subgraph Training["Training Pipeline Components"]
        MC["Megatron Core"]
        DS["DeepSpeed Integration"]
        DLS["Data Loading System"]
        OM["Optimizer Management"]
        LRS["Learning Rate Scheduling"]
    end

    subgraph Models["Model Architectures"]
        GPTX["GPT-NeoX"]
        LLAMA["LLaMA"]
        MISTRAL["Mistral"]
        RWKV["RWKV"]
        MAMBA["Mamba"]
        MOE["Mixture of Experts"]
    end

    subgraph Distributed["Distributed Infrastructure"]
        MNT["Multi-Node Training"]
        LAUNCH["Launchers"]
        COMM["GPU Communication"]
        PP["Pipeline Parallelism"]
        TP_P["Tensor Parallelism"]
        DP["Data Parallelism"]
    end

    subgraph Integration["External Integration"]
        HF["Hugging Face"]
        WB["Weights & Biases"]
        TB["TensorBoard"]
        COMET["Comet ML"]
        STORAGE["Storage Systems"]
    end

    %% Core relationships
    TP --> MA
    TP --> DTI
    TP --> DPP
    MA --> ES
    MA --> GS

    %% Training pipeline relationships
    TP --> MC
    TP --> DS
    TP --> DLS
    TP --> OM
    TP --> LRS

    %% Model relationships
    MA --> GPTX
    MA --> LLAMA
    MA --> MISTRAL
    MA --> RWKV
    MA --> MAMBA
    MA --> MOE

    %% Distributed relationships
    DTI --> MNT
    DTI --> LAUNCH
    DTI --> COMM
    DTI --> PP
    DTI --> TP_P
    DTI --> DP

    %% Support relationships
    TP --> MS
    TP --> CS
    TP --> PT
    TP --> CM

    %% Integration relationships
    MS --> WB
    MS --> TB
    MS --> COMET
    CS --> STORAGE
    MA --> HF

    %% Click events for component mapping
    click TP "https://github.com/EleutherAI/gpt-neox/blob/main/train.py"
    click OM "https://github.com/EleutherAI/gpt-neox/blob/main/megatron/optimizers.py"
    click LRS "https://github.com/EleutherAI/gpt-neox/blob/main/megatron/learning_rates.py"
    click ES "https://github.com/EleutherAI/gpt-neox/blob/main/eval.py"
    click GS "https://github.com/EleutherAI/gpt-neox/blob/main/generate.py"
    click DPP "https://github.com/EleutherAI/gpt-neox/tree/main/megatron/data/"
    click MA "https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/transformer.py"
    click MAMBA "https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/mamba/mamba.py"
    click RWKV "https://github.com/EleutherAI/gpt-neox/tree/main/megatron/model/rwkv/"
    click DTI "https://github.com/EleutherAI/gpt-neox/tree/main/megatron/mpu/"
    click CS "https://github.com/EleutherAI/gpt-neox/blob/main/megatron/checkpointing.py"
    click CM "https://github.com/EleutherAI/gpt-neox/tree/main/configs/"
    click COMET "https://github.com/EleutherAI/gpt-neox/blob/main/configs/local_setup_comet.yml"
    click WB "https://github.com/EleutherAI/gpt-neox/blob/main/configs/local_setup_wandb.yml"

    classDef core fill:#2374ab
    classDef support fill:#ff7f50
    classDef training fill:#4ca64c
    classDef models fill:#9370db
    classDef distributed fill:#daa520
    classDef integration fill:#20b2aa

    class TP,ES,GS,DPP,MA,DTI core
    class MS,CS,PT,CM support
    class MC,DS,DLS,OM,LRS training
    class GPTX,LLAMA,MISTRAL,RWKV,MAMBA,MOE models
    class MNT,LAUNCH,COMM,PP,TP_P,DP distributed
    class HF,WB,TB,COMET,STORAGE integration
```
