# apple/ml-cross-entropy architecture

> Architecture diagram of the GitHub repository apple/ml-cross-entropy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-cross-entropy
- Repository: https://github.com/apple/ml-cross-entropy

## Overview

##### Project type and purpose
- This is an open-source Python library and command-line toolkit for computing cross-entropy loss in large-vocabulary language models with minimal GPU memory usage.
- Key use cases:
– Integrate into PyTorch training loops (via linear_cross_entropy).
– Patch HuggingFace Transformers to replace standard cross-entropy.
– Run benchmarks and full model training via provided scripts.

##### File structure
- Top-level modules and directories:
– cut_cross_entropy/ : core library
– cut_cross_entropy/transformers/ : Transformers adapters
– cut_cross_entropy/vocab_parallel/ : distributed–vocabulary sharding
– benchmark/ : scripts and data for performance evaluation
– training/ : end-to-end training recipe
– scripts/ : helper shell scripts (e.g., train.sh)
– tests/ : unit tests for each kernel and utility
- Config and metadata: pyproject.toml, .pre-commit-config.yaml
- Continuous integration: tests and code style enforced via pre-commit.

##### Extract architectural insights from the README
- Core abstraction: linear_cross_entropy(embeddings, classifier, labels, …)
- Implementation strategies (strategy pattern):
– “cce”: Triton custom kernel for flash memory logits + sparse gradient filtering
– “torch_compile”: torch.compile fallback for platforms without Triton
– Variants: cce_kahan, cce_exact, etc.
- Integration extension points:
– Transformers patch via cce_patch("llama") or cce_patch(model)
– Vocabulary parallelism via VocabParallelOptions
- Benchmarking and training are invoked through Python modules (benchmark/__main__.py, training/train.py) or shell script (scripts/train.sh).

##### Main components and their interactions
a. User Application Layer
– Training code (trainer scripts, custom PyTorch loops, Transformers training API)
– Calls into cut_cross_entropy.linear_cross_entropy or applies cce_patch
b. CCE Library Layer
– Core API module (cce.py, cce_utils.py, constants.py)
– Forward kernels: cce_lse_forward.py, linear_cross_entropy.py
– Backward kernels: cce_backward.py
– Autotuning and helper: tl_autotune.py, tl_utils.py
– Implementation dispatch (strategy selector chooses Triton vs torch_compile)
c. Integration Layer
– transformers/ : model patching adapters for llama, mistral, gemma2, phi3, qwen2
– vocab_parallel/ : utilities for splitting classifier weights and distributing gradient
d. External Dependencies
– PyTorch 2.4+ (autograd, torch.compile)
– Triton 3.0+ (GPU kernels)
– CUDA hardware (Ampere or newer)
– Optional torch.distributed ProcessGroup for vocab parallelism
e. Auxiliary Modules
– benchmark/ : loads synthetic data, measures runtime and memory, reports results
– training/ : full model training pipeline, config JSON (zero3.json)
– scripts/ : convenience shell entrypoints
– tests/ : unit tests validating each component

##### Architectural patterns and design principles
- Strategy pattern for selecting loss implementation (cce vs torch_compile vs variants).
- Adapter pattern for Transformers integration (cce_patch wraps existing models).
- Modular decomposition: separate core loss logic from distribution (vocab_parallel) and adapter code.
- Single responsibility: each Python module handles a distinct piece (forward, backward, utils, constants).
- Extensibility: easy to add new impls or new model families.

##### Special notes for this library tool
- Two implementation paths (Triton vs torch.compile) and show the fallback logic.
- Optional vocabulary-parallel subprocess and how it reassembles gradients.
- How cce_patch intercepts Transformers’ forward pass to replace loss computation.
- Unit tests and benchmark runner as separate auxiliary components for development and CI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Application Layer
    subgraph "User Application Layer"
        direction TB
        Scripts["scripts/train.sh"]:::user
        Training["training/train.py"]:::user
        Benchmark["benchmark/__main__.py"]:::user
        CustomLoop["Custom PyTorch Loop"]:::user
        TransformersClient["Transformers Training API"]:::user
    end

    %% CCE Library Core
    subgraph "CCE Library Core"
        direction TB
        CoreAPI["cce.py (Core API)"]:::core
        Dispatcher["torch_compile.py (Implementation Dispatcher)"]:::core

        subgraph "Forward Kernels"
            direction TB
            LSEForward["cce_lse_forward.py"]:::core
            LinearCE["linear_cross_entropy.py"]:::core
            IndexedDot["indexed_dot.py"]:::core
        end

        Backward["cce_backward.py"]:::core

        subgraph "Constants & Utilities"
            direction TB
            Constants["constants.py"]:::core
            CCEUtils["cce_utils.py"]:::core
            Utils["utils.py"]:::core
        end

        subgraph "Autotuning & Triton Helpers"
            direction TB
            Autotune["tl_autotune.py"]:::core
            TLUtils["tl_utils.py"]:::core
        end
    end

    %% Integration Layer
    subgraph "Integration Layer"
        direction TB
        subgraph "Transformers Adapters"
            direction TB
            Patch["patch.py"]:::core
            LlamaAdapter["llama.py"]:::core
            MistralAdapter["mistral.py"]:::core
            Gemma2Adapter["gemma2.py"]:::core
            Phi3Adapter["phi3.py"]:::core
            Qwen2Adapter["qwen2.py"]:::core
            TFUtils["utils.py"]:::core
        end
        subgraph "Vocabulary Parallelism"
            direction TB
            VPUtils["utils.py (vocab_parallel)"]:::core
            VPTorchCompile["vocab_parallel_torch_compile.py"]:::core
        end
    end

    %% Auxiliary Modules
    subgraph "Auxiliary Modules"
        direction TB
        Tests["tests/... (unit tests)"]:::aux
    end

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyTorch["PyTorch 2.4+"]:::external
        Triton["Triton 3.0+"]:::external
        CUDA["CUDA GPU"]:::external
        ProcessGroup["torch.distributed ProcessGroup"]:::external
        GPUMem["GPU Memory (flash + global)"]:::external
    end

    %% Connections
    Scripts -->|"runs benchmark/train"| CoreAPI
    Training -->|"invokes loss"| CoreAPI
    Benchmark -->|"calls loss"| CoreAPI
    CustomLoop -->|"calls linear_cross_entropy"| CoreAPI
    TransformersClient -->|"forward pass"| Patch
    Patch -->|"wraps loss call"| CoreAPI

    CoreAPI -->|"select impl"| Dispatcher
    Dispatcher -->|"Triton path"| LSEForward
    Dispatcher -->|"Triton path"| LinearCE
    Dispatcher -->|"Triton path"| IndexedDot
    Dispatcher -->|"Fallback"| DispatchTorchCompile --> TLCompile["torch_compile path"]:::core
    TLCompile -->|"compute via torch.compile"| GPUMem

    LSEForward -->|"embeddings, logits"| GPUMem
    LinearCE -->|"embeddings, logits"| GPUMem
    IndexedDot -->|"embeddings × weights"| GPUMem
    Backward -->|"gradients"| GPUMem

    CoreAPI -->|"constants & helpers"| Constants
    CoreAPI -->|"utils"| CCEUtils
    CoreAPI -->|"general utils"| Utils

    LSEForward --> Autotune
    LSEForward --> TLUtils

    CoreAPI -->|"adapter extension"| Patch
    CoreAPI -->|"vocab parallel options"| VPUtils
    VPUtils --> VPTorchCompile
    VPTorchCompile --> ProcessGroup
    ProcessGroup -->|"sharded weights"| GPUMem

    Tests --> CoreAPI
    Tests --> LSEForward
    Tests --> Backward
    Tests --> IndexedDot
    Tests --> VPUtils

    %% Click Events
    click CoreAPI "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/cce.py"
    click LSEForward "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/cce_lse_forward.py"
    click LinearCE "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/linear_cross_entropy.py"
    click IndexedDot "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/indexed_dot.py"
    click Backward "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/cce_backward.py"
    click Constants "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/constants.py"
    click CCEUtils "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/cce_utils.py"
    click Utils "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/utils.py"
    click Autotune "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/tl_autotune.py"
    click TLUtils "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/tl_utils.py"
    click TLCompile "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/torch_compile.py"
    click Patch "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/patch.py"
    click LlamaAdapter "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/llama.py"
    click MistralAdapter "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/mistral.py"
    click Gemma2Adapter "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/gemma2.py"
    click Phi3Adapter "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/phi3.py"
    click Qwen2Adapter "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/qwen2.py"
    click TFUtils "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/transformers/utils.py"
    click VPUtils "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/vocab_parallel/utils.py"
    click VPTorchCompile "https://github.com/apple/ml-cross-entropy/blob/main/cut_cross_entropy/vocab_parallel/vocab_parallel_torch_compile.py"
    click Benchmark "https://github.com/apple/ml-cross-entropy/blob/main/benchmark/__main__.py"
    click Scripts "https://github.com/apple/ml-cross-entropy/blob/main/scripts/train.sh"
    click Training "https://github.com/apple/ml-cross-entropy/blob/main/training/train.py"

    %% Styles
    classDef user fill:#e0f7e9,stroke:#2d9a4e,color:#2d9a4e;
    classDef core fill:#e1ecf4,stroke:#0b5394,color:#0b5394;
    classDef external fill:#ffe6cc,stroke:#d35400,color:#d35400;
    classDef aux fill:#f3e6ff,stroke:#8e44ad,color:#8e44ad;
```
