# nvidia-merlin/dataloader architecture

> Architecture diagram of the GitHub repository nvidia-merlin/dataloader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia-merlin/dataloader
- Repository: https://github.com/nvidia-merlin/dataloader

## Overview

##### Project type and purpose
- This is a Python library (“merlin-dataloader”) for high-performance GPU-accelerated data loading in recommender-system training.
- It provides zero-copy transfers from disk (Parquet files) into GPU memory and then into TensorFlow, PyTorch, or JAX via DLPack.
- Its core use-case is large-scale, distributed recommender-model training with per-epoch shuffling, embedding support, padding, and multi-GPU/distributed backends.

##### Top-level structure and architectural cues
- merlin/dataloader: core operators, versioning, and framework-agnostic loader base.
- merlin/dataloader/ops: GPU-optimized ops (embeddings, padding).
- merlin/dataloader/utils: framework-specific trainers and wrappers (TensorFlow Horovod, Torch Distributed).
- merlin/loader: thin framework adapters for JAX, TensorFlow, PyTorch.
- examples: usage notebooks showing end-to-end flow.
- docs: Sphinx-based documentation.
- tests: unit and integration tests against framework loaders and examples.

##### Key README insights
- Inputs: on-disk Parquet dataset → merlin.io.Dataset (not part of this repo, but assumed upstream).
- Loader instantiates with batch_size and framework choice.
- Core flow: read Parquet → GPU buffer → DLPack → framework tensors → model.fit/loop.

##### Main components
a. Data Source
- Parquet files on disk (HDFS/S3/local FS)
b. Data Reader & Buffering
- merlin.dataloader: reads batches with GPU-side reader (e.g., cuDF or custom reader)
- In-GPU staging buffers
c. Data-Processing Ops
- Embedding lookups, padding, shuffling
- Operate on GPU arrays
d. DLPack Interface
- Zero-copy bridge to framework tensors
e. Framework Loaders
- LoaderTensorFlow: tf.data.Iterator or Keras Sequence
- LoaderTorch: torch.utils.data.DataLoader subclass with distributed support
- LoaderJAX: iterator yielding jax.numpy arrays
f. External Services & Dependencies
- CUDA Runtime, cuDF (if used), DLPack protocol
- Horovod or torch.distributed for distributed training
g. Consumer
- User model training loop (model.fit, training script)

##### Architectural patterns and design principles
- Plugin/adapter pattern for framework neutrality (loader_base + per-framework implementations).
- Single Responsibility: separate modules for I/O, ops, adapters.
- Zero-copy and in-GPU data movement to minimize CPU ↔ GPU overhead.
- Extensibility: ops directory can be extended with new GPU operations.
- Modular testing: unit tests for each loader and op.

##### Details to capture
- GPU buffer staging between Reader and Ops.
- DLPack zero-copy transfer boundary.
- The three framework loader paths in parallel.
- Points of extensibility (ops plugins, new frameworks).
- Distributed-training hooks (Horovod/TorchDist).
- Distinguished modules: merlin.dataloader vs merlin.loader.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        Parquet["Parquet Files (HDFS/S3/FS)"]:::storage
    end

    %% Core Library
    subgraph "Core Library (merlin.dataloader)" 
        direction TB
        Version["_version.py"]:::core
        LoaderBase["loader_base.py"]:::core
        subgraph "GPU Pipeline"
            direction TB
            GPUBuffer["GPU Staging Buffer"]:::core
            subgraph "Data-Processing Ops"
                direction TB
                Embeddings["embeddings.py"]:::core
                Padding["padding.py"]:::core
            end
        end
        DLPackUtils["tf_utils.py"]:::core
        JAXLoader["jax.py"]:::core
        TFLoader["tensorflow.py"]:::core
        TorchLoader["torch.py"]:::core
    end

    %% Adapter Layer
    subgraph "Framework Adapters (merlin.loader)"
        direction TB
        AdapterJAX["jax.py"]:::adapter
        AdapterTF["tensorflow.py"]:::adapter
        AdapterTorch["torch.py"]:::adapter
    end

    %% External Services
    subgraph "External Services & Dependencies"
        direction TB
        CUDA["CUDA Runtime"]:::external
        cuDF["cuDF"]:::external
        DLPackProto["DLPack Protocol"]:::external
        Horovod["Horovod"]:::external
        TorchDist["torch.distributed"]:::external
    end

    %% User Code
    subgraph "User Model Training"
        direction TB
        TFExample["01a-Getting-started-Tensorflow.ipynb"]:::user
        TorchExample["01b-Getting-started-Pytorch.ipynb"]:::user
        MultiGPUExample["02-Multi-GPU-Tensorflow-with-Horovod.ipynb"]:::user
        ModelLoop["Model Training Script"]:::user
    end

    %% Connections
    Parquet -->|"1 Read batches"| LoaderBase
    LoaderBase -->|"2 Stage data"| GPUBuffer
    GPUBuffer -->|"3 Apply ops"| Embeddings
    GPUBuffer --> Embeddings
    GPUBuffer --> Padding
    Embeddings -->|"4 To DLPack"| DLPackUtils
    Padding --> DLPackUtils
    DLPackUtils -->|"5 Adapter"| AdapterTF
    DLPackUtils --> AdapterTorch
    DLPackUtils --> AdapterJAX
    AdapterTF -->|"6 Train"| ModelLoop
    AdapterTorch --> ModelLoop
    AdapterJAX --> ModelLoop

    %% Distributed hooks
    AdapterTF --> Horovod
    AdapterTorch --> TorchDist

    %% Click Events
    click Version "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/_version.py"
    click LoaderBase "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/loader_base.py"
    click Embeddings "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/ops/embeddings.py"
    click Padding "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/ops/padding.py"
    click DLPackUtils "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/tf_utils.py"
    click JAXLoader "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/jax.py"
    click TFLoader "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/tensorflow.py"
    click TorchLoader "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/dataloader/torch.py"
    click AdapterJAX "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/loader/jax.py"
    click AdapterTF "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/loader/tensorflow.py"
    click AdapterTorch "https://github.com/nvidia-merlin/dataloader/blob/main/merlin/loader/torch.py"
    click TFExample "https://github.com/nvidia-merlin/dataloader/blob/main/examples/01a-Getting-started-Tensorflow.ipynb"
    click TorchExample "https://github.com/nvidia-merlin/dataloader/blob/main/examples/01b-Getting-started-Pytorch.ipynb"
    click MultiGPUExample "https://github.com/nvidia-merlin/dataloader/blob/main/examples/02-Multi-GPU-Tensorflow-with-Horovod.ipynb"

    %% Styles
    classDef storage fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef core fill:#cce5ff,stroke:#0366d6,stroke-width:1px
    classDef adapter fill:#d4edda,stroke:#28a745,stroke-width:1px
    classDef external fill:#ffe5b4,stroke:#e69500,stroke-width:1px
    classDef user fill:#f2dede,stroke:#d32f2f,stroke-width:1px
```
