# ml-explore/mlx-data architecture

> Architecture diagram of the GitHub repository ml-explore/mlx-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ml-explore/mlx-data
- Repository: https://github.com/ml-explore/mlx-data

## Overview

##### Project type and purpose
- This is an open-source, framework-agnostic data-loading library (MLX Data) written in C++ with Python bindings.
- Purpose: Provide a high-performance, flexible data pipeline for machine-learning workloads (PyTorch, Jax, MLX), exposing a fluent API to construct data buffers, streams, transformations, and batching.

##### File structure
- Top-level:
- cmake/ & super/: CMake modules and patches for third-party libs
- mlx/data/: Core C++ library subdivided into:
– data (Array, Dataset, Stream, Sample abstractions)
– data/buffer (batching primitives, shuffling, partitioning)
– data/core (I/O: CSV, TAR, AWS fetcher, tokenizers, thread pool)
– data/op (element-wise and sample-wise operations)
– data/stream (streaming abstractions with prefetch, repeat, sliding window)
- python/src & python/mlx/: PyBind11 wrappers and the Python package
- benchmarks/: Comparative benchmark scripts vs PyTorch/TFDS on datasets
- docs/: Sphinx documentation source
- tests/: C++ and Python tests
- build scripts and config: CMakeLists.txt, pyproject.toml, setup.py, CI config

##### README for additional insights
- Describes Python and C++ APIs for building pipelines: buffer → stream → transforms → batch → prefetch
- Shows integration examples with Python functions, illustrating extensibility

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    subgraph "User-Facing API" 
        PythonAPI["Python API (mlx.data)"]:::userapi
    end

    subgraph "Binding Layer"
        Binding["PyBind11 Wrapper (wrap.cpp, wrap_*.cpp)"]:::engine
    end

    subgraph "C++ Core Library" 
        subgraph "Data Abstractions" 
            DataAbstractions["Dataset, Array, Stream, Sample"]:::engine
        end
        subgraph "Buffer Modules" 
            BufferModules["Batch, Shuffle, Partition, DynamicBatch, Perm, Transform"]:::engine
        end
        subgraph "I/O & Utilities" 
            CoreIO["CSVReader, TARReader, AWSFileFetcher, Tokenizers, ThreadPool, Utils"]:::engine
        end
        subgraph "Operation Modules" 
            OpModules["LoadImage/Audio/Video, Pad, Tokenize, Filter, Rename, etc."]:::engine
        end
        subgraph "Streaming Modules" 
            StreamModules["Prefetch, OrderedPrefetch, Repeat, SlidingWindow, Batch, Transform"]:::engine
        end
    end

    subgraph "External Dependencies"
        FS["File System / AWS S3"]:::ext
        Libs["FFMPEG, libsndfile, JPEG Turbo"]:::ext
        MLFW["PyTorch, TFDS, Jax"]:::ext
    end

    subgraph "Infrastructure"
        BuildSys["Build System & Patches"]:::infra
        PackagingCI["Python Packaging & CI"]:::infra
        Docs["Documentation (Sphinx)"]:::infra
        Benchmarks["Benchmarks & Examples"]:::infra
    end

    %% Data Flow
    PythonAPI -->|"API Calls"| Binding
    Binding -->|"C++ Calls"| DataAbstractions
    DataAbstractions -->|"provides Samples to"| BufferModules
    BufferModules -->|"outputs Data to"| StreamModules
    StreamModules -->|"pipes Data through"| OpModules
    OpModules -->|"feeds Data into"| BufferModules
    StreamModules -->|"batches Data via"| BufferModules
    StreamModules -->|"prefetches via"| StreamModules
    CoreIO -->|"I/O Services"| DataAbstractions
    CoreIO -->|"uses"| Libs
    DataAbstractions -->|"reads/writes"| FS

    PythonAPI -.->|"integrates"| MLFW
    Benchmarks -->|"tests"| PythonAPI
    Benchmarks -->|"tests"| MLFW

    BuildSys --> Binding
    BuildSys --> DataAbstractions
    BuildSys --> BufferModules
    BuildSys --> CoreIO
    BuildSys --> OpModules
    BuildSys --> StreamModules

    PackagingCI --> PythonAPI
    PackagingCI --> Binding

    Docs --> PythonAPI
    Docs --> Binding
    Docs --> DataAbstractions
    Docs --> BufferModules
    Docs --> CoreIO
    Docs --> OpModules
    Docs --> StreamModules

    %% Click Events
    click PythonAPI "https://github.com/ml-explore/mlx-data/blob/main/python/mlx/data/__init__.py"
    click PythonAPI "https://github.com/ml-explore/mlx-data/tree/main/python/mlx/data/"
    click Binding "https://github.com/ml-explore/mlx-data/tree/main/python/src/"
    click DataAbstractions "https://github.com/ml-explore/mlx-data/blob/main/mlx/data/Array.cpp"
    click BufferModules "https://github.com/ml-explore/mlx-data/tree/main/mlx/data/buffer/"
    click CoreIO "https://github.com/ml-explore/mlx-data/tree/main/mlx/data/core/"
    click OpModules "https://github.com/ml-explore/mlx-data/tree/main/mlx/data/op/"
    click StreamModules "https://github.com/ml-explore/mlx-data/tree/main/mlx/data/stream/"
    click BuildSys "https://github.com/ml-explore/mlx-data/blob/main/CMakeLists.txt"
    click BuildSys "https://github.com/ml-explore/mlx-data/tree/main/cmake/"
    click PackagingCI "https://github.com/ml-explore/mlx-data/blob/main/setup.py"
    click PackagingCI "https://github.com/ml-explore/mlx-data/blob/main/pyproject.toml"
    click PackagingCI "https://github.com/ml-explore/mlx-data/blob/main/.circleci/config.yml"
    click Docs "https://github.com/ml-explore/mlx-data/tree/main/docs/src/"
    click Benchmarks "https://github.com/ml-explore/mlx-data/tree/main/benchmarks/comparative/"

    %% Styles
    classDef userapi fill:#9f6,stroke:#333,stroke-width:1px
    classDef engine fill:#69f,stroke:#333,stroke-width:1px
    classDef ext fill:#f96,stroke:#333,stroke-width:1px
    classDef infra fill:#999,stroke:#333,stroke-width:1px
```
