How xuezhemax/megalodon works
Project Type and Purpose
• Megalodon is a research‐grade Python library for pretraining and inference of a large language model (LLM) called “Megalodon 7B.”
• It combines high‐level Python orchestration (data loading, training loop, distributed setup) with low‐level C++/CUDA extensions for performance‐critical kernels (attention, FFT convolution, normalization).
• The system supports unlimited context length via chunked model parallelism and is designed for large‐scale distributed GPU training and evaluation.
High-Level Architectural Components
a. Python Orchestration Layer
• Entry points: eval.py (inference/evaluation), training scripts or user code that imports megalodon/logger.py, megalodon/distributed/*, megalodon/model/mega.py, megalodon/optim, etc.
• Responsibilities: parse CLI args, initialize logger, set up environment, call distributed init, build model, run forward/backward loops, manage checkpoints and metrics.
b. Core Library Package (megalodon/)
• Configuration (config.py): holds hyperparameters, parallelism settings.
• Data (data/): dataloader.py and tokenizer.py for preparing JSONL text datasets.
• Model Definition (model/mega.py): Python‐level model builder wrapping Megatron‐style Transformer blocks.
• Modules (modules/): high‐level PyTorch modules that call fused/CUDA kernels (attention, EMA, FFT conv, norms), chunk parallel layers, model parallel layers.
• Optimization (optim/): learning‐rate scheduling (lr_scheduler.py), optimizer wrapper (optimizer.py).
• Utilities (utils.py, logger.py, reloading.py): environment setup, seed, logging, checkpoint I/O.
c. C++/CUDA Extension Layer (csrc/)
• Custom CUDA kernels for heavy operations: attention_kernel.cu, fftconv_kernel.cu, sequence_norm_kernel.cu, ema_*_kernel.cu, etc.
• Host–device wrappers (*.cc, *.h, *.cuh): blas, random_utils, cuda_utils, kernel registration.
• Built via setuptools in setup.py, exposed to Python under megalodon.csrc and bridged by custom PyTorch ops.
d. Distributed Infrastructure (distributed/)
• Initialization (initialize.py, setup.py): start torch.distributed, configure NCCL, handle SLURM.
• Parallelism strategies: data parallel, model parallel, chunk parallel (fairscale_fsdp.py for FSDP integration, utility functions).
• And rank management across GPU clusters.
External Dependencies and Services
• PyTorch 2.0.1 with CUDA support
• NVIDIA Apex (for mixed‐precision and fused ops)
• FairScale (for Fully Sharded Data Parallel)
• CUDA runtime/drivers
• SLURM or other cluster scheduler for job orchestration
Relationships and Data Flow
• User Script (e.g., eval.py or training entry) → calls logger/env setup → distributed init
• Distributed init → spawns processes, sets global and local ranks, builds process groups
• DataLoader → yields tokenized batches → fed into Python model modules
• Python Model Modules → invoke fused CUDA ops via C++/CUDA extension → return tensors
• Training loop → loss computation → backward pass → model.grad_all_reduce() → optimizer step → scheduler step
• Checkpointing and logging happen through utility modules
Architectural Patterns and Design Principles
• Layered separation: Python orchestration → modular library → C++/CUDA kernels
• Modular parallelism: explicit support for data, model, chunk, and FSDP sharding
• High cohesion: each directory encapsulates a single concern (data, model, ops, distributed)
• Performance via fused operations and custom kernels to minimize Python/CUDA overhead
• Extensibility: new kernels or parallel strategies can be added by extending modules/ and csrc/
Elements
• Clear labels: “DataLoader,” “Model Builder,” “Fused Attention Kernel,” “Optimizer/Scheduler,” “Torch Distributed Init”
• Parallel group visualization: annotate which components are sharded or replicated across GPUs
• Version and context: note PyTorch 2.0.1 + CUDA 11.7, Apex v23.08, FairScale branch