How apple/ml-cross-entropy works
Project type and purpose
• This is an open-source Python library and command-line toolkit for computing cross-entropy loss in large-vocabulary language models with minimal GPU memory usage.
• Key use cases:
– Integrate into PyTorch training loops (via linear_cross_entropy).
– Patch HuggingFace Transformers to replace standard cross-entropy.
– Run benchmarks and full model training via provided scripts.
File structure
• Top-level modules and directories:
– cut_cross_entropy/ : core library
– cut_cross_entropy/transformers/ : Transformers adapters
– cut_cross_entropy/vocab_parallel/ : distributed–vocabulary sharding
– benchmark/ : scripts and data for performance evaluation
– training/ : end-to-end training recipe
– scripts/ : helper shell scripts (e.g., train.sh)
– tests/ : unit tests for each kernel and utility
• Config and metadata: pyproject.toml, .pre-commit-config.yaml
• Continuous integration: tests and code style enforced via pre-commit.
Extract architectural insights from the README
• Core abstraction: linear_cross_entropy(embeddings, classifier, labels, …)
• Implementation strategies (strategy pattern):
– “cce”: Triton custom kernel for flash memory logits + sparse gradient filtering
– “torch_compile”: torch.compile fallback for platforms without Triton
– Variants: cce_kahan, cce_exact, etc.
• Integration extension points:
– Transformers patch via cce_patch("llama") or cce_patch(model)
– Vocabulary parallelism via VocabParallelOptions
• Benchmarking and training are invoked through Python modules (benchmark/__main__.py, training/train.py) or shell script (scripts/train.sh).
Main components and their interactions
a. User Application Layer
– Training code (trainer scripts, custom PyTorch loops, Transformers training API)
– Calls into cut_cross_entropy.linear_cross_entropy or applies cce_patch
b. CCE Library Layer
– Core API module (cce.py, cce_utils.py, constants.py)
– Forward kernels: cce_lse_forward.py, linear_cross_entropy.py
– Backward kernels: cce_backward.py
– Autotuning and helper: tl_autotune.py, tl_utils.py
– Implementation dispatch (strategy selector chooses Triton vs torch_compile)
c. Integration Layer
– transformers/ : model patching adapters for llama, mistral, gemma2, phi3, qwen2
– vocab_parallel/ : utilities for splitting classifier weights and distributing gradient
d. External Dependencies
– PyTorch 2.4+ (autograd, torch.compile)
– Triton 3.0+ (GPU kernels)
– CUDA hardware (Ampere or newer)
– Optional torch.distributed ProcessGroup for vocab parallelism
e. Auxiliary Modules
– benchmark/ : loads synthetic data, measures runtime and memory, reports results
– training/ : full model training pipeline, config JSON (zero3.json)
– scripts/ : convenience shell entrypoints
– tests/ : unit tests validating each component
Architectural patterns and design principles
• Strategy pattern for selecting loss implementation (cce vs torch_compile vs variants).
• Adapter pattern for Transformers integration (cce_patch wraps existing models).
• Modular decomposition: separate core loss logic from distribution (vocab_parallel) and adapter code.
• Single responsibility: each Python module handles a distinct piece (forward, backward, utils, constants).
• Extensibility: easy to add new impls or new model families.
Special notes for this library tool
• Two implementation paths (Triton vs torch.compile) and show the fallback logic.
• Optional vocabulary-parallel subprocess and how it reassembles gradients.
• How cce_patch intercepts Transformers’ forward pass to replace loss computation.
• Unit tests and benchmark runner as separate auxiliary components for development and CI.