# xuezhemax/megalodon architecture

> Architecture diagram of the GitHub repository xuezhemax/megalodon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xuezhemax/megalodon
- Repository: https://github.com/xuezhemax/megalodon

## Overview

##### Project Type and Purpose
- Megalodon is a research‐grade Python library for pretraining and inference of a large language model (LLM) called “Megalodon 7B.”
- It combines high‐level Python orchestration (data loading, training loop, distributed setup) with low‐level C++/CUDA extensions for performance‐critical kernels (attention, FFT convolution, normalization).
- The system supports unlimited context length via chunked model parallelism and is designed for large‐scale distributed GPU training and evaluation.

##### High-Level Architectural Components
a. Python Orchestration Layer
- Entry points: `eval.py` (inference/evaluation), training scripts or user code that imports `megalodon/logger.py`, `megalodon/distributed/*`, `megalodon/model/mega.py`, `megalodon/optim`, etc.
- Responsibilities: parse CLI args, initialize logger, set up environment, call distributed init, build model, run forward/backward loops, manage checkpoints and metrics.
b. Core Library Package (`megalodon/`)
- Configuration (`config.py`): holds hyperparameters, parallelism settings.
- Data (`data/`): `dataloader.py` and `tokenizer.py` for preparing JSONL text datasets.
- Model Definition (`model/mega.py`): Python‐level model builder wrapping Megatron‐style Transformer blocks.
- Modules (`modules/`): high‐level PyTorch modules that call fused/CUDA kernels (attention, EMA, FFT conv, norms), chunk parallel layers, model parallel layers.
- Optimization (`optim/`): learning‐rate scheduling (`lr_scheduler.py`), optimizer wrapper (`optimizer.py`).
- Utilities (`utils.py`, `logger.py`, `reloading.py`): environment setup, seed, logging, checkpoint I/O.
c. C++/CUDA Extension Layer (`csrc/`)
- Custom CUDA kernels for heavy operations: `attention_kernel.cu`, `fftconv_kernel.cu`, `sequence_norm_kernel.cu`, `ema_*_kernel.cu`, etc.
- Host–device wrappers (`*.cc`, `*.h`, `*.cuh`): `blas`, `random_utils`, `cuda_utils`, kernel registration.
- Built via setuptools in `setup.py`, exposed to Python under `megalodon.csrc` and bridged by custom PyTorch ops.
d. Distributed Infrastructure (`distributed/`)
- Initialization (`initialize.py`, `setup.py`): start torch.distributed, configure NCCL, handle SLURM.
- Parallelism strategies: data parallel, model parallel, chunk parallel (`fairscale_fsdp.py` for FSDP integration, utility functions).
- And rank management across GPU clusters.

##### External Dependencies and Services
- PyTorch 2.0.1 with CUDA support
- NVIDIA Apex (for mixed‐precision and fused ops)
- FairScale (for Fully Sharded Data Parallel)
- CUDA runtime/drivers
- SLURM or other cluster scheduler for job orchestration

##### Relationships and Data Flow
- User Script (e.g., `eval.py` or training entry) → calls logger/env setup → distributed init
- Distributed init → spawns processes, sets global and local ranks, builds process groups
- DataLoader → yields tokenized batches → fed into Python model modules
- Python Model Modules → invoke fused CUDA ops via C++/CUDA extension → return tensors
- Training loop → loss computation → backward pass → `model.grad_all_reduce()` → optimizer step → scheduler step
- Checkpointing and logging happen through utility modules

##### Architectural Patterns and Design Principles
- Layered separation: Python orchestration → modular library → C++/CUDA kernels
- Modular parallelism: explicit support for data, model, chunk, and FSDP sharding
- High cohesion: each directory encapsulates a single concern (data, model, ops, distributed)
- Performance via fused operations and custom kernels to minimize Python/CUDA overhead
- Extensibility: new kernels or parallel strategies can be added by extending `modules/` and `csrc/`

##### Elements
- Clear labels: “DataLoader,” “Model Builder,” “Fused Attention Kernel,” “Optimizer/Scheduler,” “Torch Distributed Init”
- Parallel group visualization: annotate which components are sharded or replicated across GPUs
- Version and context: note PyTorch 2.0.1 + CUDA 11.7, Apex v23.08, FairScale branch

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry & CLI
    subgraph "1. Entry & CLI"
        Eval["eval.py"]:::python
        Generation["generation.py"]:::python
        Inference["inference.py"]:::python
    end

    %% Python Orchestration Layer
    subgraph "2. Python Orchestration Layer"
        Logger["logger.py"]:::python
        Utils["utils.py"]:::python
        Reloading["reloading.py"]:::python
        Config["config.py"]:::python
    end

    %% Data Ingestion
    subgraph "3. Data Ingestion"
        DataLoader["dataloader.py"]:::python
        Tokenizer["tokenizer.py"]:::python
    end

    %% Model Definition
    subgraph "4. Model Definition / Builder"
        ModelBuilder["mega.py"]:::python
    end

    %% Modules
    subgraph "5. High-Level Modules"
        AttentionSoftmax["attention_softmax.py"]:::python
        FusedAttention["fused_attention.py"]:::python
        EMAComplex["complex_exponential_moving_average.py"]:::python
        LayerNorm["layer_norm.py"]:::python
        Losses["losses.py"]:::python
        MAGAttention["moving_average_gated_attention.py"]:::python
        FFN["normalized_feedforward_network.py"]:::python
        RotaryPE["rotary_positional_embedding.py"]:::python
        TimestepNorm["timestep_norm.py"]:::python
        subgraph "Chunk-Parallel Layers"
            ChunkMap["mappings.py"]:::python
        end
        subgraph "Model-Parallel Layers"
            MPLayers["layers.py"]:::python
            MPMap["mappings.py"]:::python
        end
        subgraph "Fused Ops (Python Wrappers)"
            SoftmaxOp["softmax.py"]:::python
            SwiftOp["swift.py"]:::python
            EMAHiddenOp["ema_hidden.py"]:::python
            EMAParamOp["ema_parameters.py"]:::python
            FFTConvOp["fftconv.py"]:::python
            MemEffDrop["memory_efficient_dropout.py"]:::python
        end
    end

    %% Optimization
    subgraph "6. Optimization"
        Scheduler["lr_scheduler.py"]:::python
        Optimizer["optimizer.py"]:::python
    end

    %% Distributed Infrastructure
    subgraph "7. Distributed Infrastructure"
        InitDist["initialize.py"]:::python
        SetupDist["setup.py"]:::python
        FSDP["fairscale_fsdp.py"]:::python
        DistUtils["utils.py"]:::python
    end

    %% C++/CUDA Extension Layer
    subgraph "8. C++/CUDA Extension"
        ExtModule["megalodon_extension.cc"]:::cuda
        subgraph "Core Wrappers & Utils"
            BlasCC["blas.cc"]:::cuda
            BlasH["blas.h"]:::cuda
            ComplexUtils["complex_utils.cuh"]:::cuda
            CudaUtils["cuda_utils.cuh"]:::cuda
            FFTUtils["fft.cuh"]:::cuda
        end
        subgraph "Ops & Kernels"
            AttCC["attention.cc"]:::cuda
            AttKernel["attention_kernel.cu"]:::cuda
            SoftmaxCC["attention_softmax.cc"]:::cuda
            SoftmaxKer["attention_softmax_kernel.cu"]:::cuda
            EMAHiddenCC["ema_hidden.cc"]:::cuda
            EMAHiddenKer["ema_hidden_kernel.cu"]:::cuda
            EMAParamCC["ema_parameters.cc"]:::cuda
            EMAParamKer["ema_parameters_kernel.cu"]:::cuda
            FFTConvCC["fftconv.cc"]:::cuda
            FFTConvKer["fftconv_kernel.cu"]:::cuda
            SeqNormCC["sequence_norm.cc"]:::cuda
            SeqNormKer["sequence_norm_kernel.cu"]:::cuda
            TSNormCC["timestep_norm.cc"]:::cuda
            TSNormKer["timestep_norm_kernel.cu"]:::cuda
        end
    end

    %% External Dependencies
    subgraph "9. External Dependencies"
        PyTorch["PyTorch 2.0.1 + CUDA"]:::external
        Apex["NVIDIA Apex"]:::external
        FairScale["FairScale"]:::external
        CUDA["CUDA Runtime/Driver"]:::external
        SLURM["Cluster Scheduler (SLURM)"]:::external
    end

    %% Workflow Arrows
    Eval -->|calls| Logger -->|init| InitDist --> SetupDist
    SetupDist --> FSDP
    Eval --> ModelBuilder
    ModelBuilder --> AttentionSoftmax
    AttentionSoftmax --> SoftmaxOp --> SoftmaxCC --> SoftmaxKer --> SoftmaxOp
    ModelBuilder --> FusedAttention --> AttCC --> AttKernel --> FusedAttention
    DataLoader --> Tokenizer --> ModelBuilder
    ModelBuilder --> LossComp["Loss Computation"]:::python
    LossComp --> Backprop["Backward Pass"]:::python --> GradAllReduce["grad_all_reduce()"]:::python
    GradAllReduce --> Optimizer --> Scheduler
    Scheduler --> Checkpoint["Checkpoint I/O"]:::python -->|uses| Utils

    %% Click Events
    click Eval "https://github.com/xuezhemax/megalodon/blob/main/eval.py"
    click Generation "https://github.com/xuezhemax/megalodon/blob/main/megalodon/generation.py"
    click Inference "https://github.com/xuezhemax/megalodon/blob/main/megalodon/inference.py"
    click Logger "https://github.com/xuezhemax/megalodon/blob/main/megalodon/logger.py"
    click Utils "https://github.com/xuezhemax/megalodon/blob/main/megalodon/utils.py"
    click Reloading "https://github.com/xuezhemax/megalodon/blob/main/megalodon/reloading.py"
    click Config "https://github.com/xuezhemax/megalodon/blob/main/megalodon/config.py"
    click DataLoader "https://github.com/xuezhemax/megalodon/blob/main/megalodon/data/dataloader.py"
    click Tokenizer "https://github.com/xuezhemax/megalodon/blob/main/megalodon/data/tokenizer.py"
    click ModelBuilder "https://github.com/xuezhemax/megalodon/blob/main/megalodon/model/mega.py"
    click AttentionSoftmax "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/attention_softmax.py"
    click FusedAttention "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_attention.py"
    click EMAComplex "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/complex_exponential_moving_average.py"
    click LayerNorm "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/layer_norm.py"
    click Losses "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/losses.py"
    click MAGAttention "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/moving_average_gated_attention.py"
    click FFN "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/normalized_feedforward_network.py"
    click RotaryPE "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/rotary_positional_embedding.py"
    click TimestepNorm "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/timestep_norm.py"
    click ChunkMap "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/chunk_parallel/mappings.py"
    click MPLayers "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/model_parallel/layers.py"
    click MPMap "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/model_parallel/mappings.py"
    click SoftmaxOp "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/attention/softmax.py"
    click SwiftOp "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/attention/swift.py"
    click EMAHiddenOp "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/ema_hidden.py"
    click EMAParamOp "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/ema_parameters.py"
    click FFTConvOp "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/fftconv.py"
    click MemEffDrop "https://github.com/xuezhemax/megalodon/blob/main/megalodon/modules/fused_ops/memory_efficient_dropout.py"
    click Scheduler "https://github.com/xuezhemax/megalodon/blob/main/megalodon/optim/lr_scheduler.py"
    click Optimizer "https://github.com/xuezhemax/megalodon/blob/main/megalodon/optim/optimizer.py"
    click InitDist "https://github.com/xuezhemax/megalodon/blob/main/megalodon/distributed/initialize.py"
    click SetupDist "https://github.com/xuezhemax/megalodon/blob/main/megalodon/distributed/setup.py"
    click FSDP "https://github.com/xuezhemax/megalodon/blob/main/megalodon/distributed/fairscale_fsdp.py"
    click DistUtils "https://github.com/xuezhemax/megalodon/blob/main/megalodon/distributed/utils.py"
    click ExtModule "https://github.com/xuezhemax/megalodon/blob/main/megalodon/csrc/megalodon_extension.cc"

    %% Styles
    classDef python fill:#D0E8FF,stroke:#0288D1,color:#000
    classDef cuda fill:#FFE0B2,stroke:#FB8C00,color:#000
    classDef external fill:#C8E6C9,stroke:#388E3C,color:#000
```
