# huggingface/datasets architecture

> Architecture diagram of the GitHub repository huggingface/datasets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/datasets
- Repository: https://github.com/huggingface/datasets

## Overview

🤗 Datasets loads public Hub datasets or local files, optionally streams them, builds typed Arrow-backed datasets, and exposes them for preprocessing and model use. The core path runs from the user through load_dataset and a format-specific builder, with download/streaming support and Arrow storage; Dataset operations then transform data, and formatters decode it for consumers. Search indexes, Hub publishing, and the broad set of packaged formats and multimodal features are additional capabilities. The graph reflects documented capabilities where implementation excerpts were unavailable; those links are omitted when call ownership is uncertain.

Main entry points: [`src/datasets/arrow_dataset.py`](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py) (Dataset API), [`src/datasets/builder.py`](https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py) (Builder lifecycle), [`src/datasets/load.py`](https://github.com/huggingface/datasets/blob/main/src/datasets/load.py) (Load API), [`src/datasets/download/download_manager.py`](https://github.com/huggingface/datasets/blob/main/src/datasets/download/download_manager.py) (Download manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_load["Loading and sources"]
  node_loadapi["Load API<br/>[load.py]"]
  node_builder["Builder lifecycle<br/>[builder.py]"]
  node_formats["Packaged builders"]
  node_download["Download manager"]
  node_streamdownload["Streaming manager"]
  node_hub["Hub access<br/>[hub.py]"]
end

subgraph group_core["Dataset core"]
  node_dataset["Dataset API<br/>[arrow_dataset.py]"]
  node_iterable["Iterable dataset"]
  node_datasetdict["Dataset splits<br/>[dataset_dict.py]"]
  node_arrow[("Arrow tables<br/>[table.py]")]
  node_features["Typed features<br/>[features.py]"]
  node_cache[("Cache fingerprints<br/>[fingerprint.py]")]
end

subgraph group_access["Processing and access"]
  node_transform["Dataset transforms<br/>[arrow_dataset.py]"]
  node_parallel["Parallel processing<br/>[parallel.py]"]
  node_formatting["Output formatting<br/>[formatting.py]"]
  node_search["Search indexes<br/>[search.py]"]
end

subgraph group_integrations["Interoperability and sharing"]
  node_frameworks["ML frameworks"]
  node_io["Import/export"]
  node_storage["Storage buckets"]
end

node_user(("Library user"))
node_externalhub["Hugging Face Hub"]

node_user -->|"requests dataset"| node_loadapi
node_loadapi -->|"creates builder"| node_builder
node_builder -->|"selects format"| node_formats
node_builder -->|"downloads files"| node_download
node_builder -.->|"prepares stream"| node_streamdownload
node_loadapi -->|"resolves Hub data"| node_hub
node_hub -->|"queries repository"| node_externalhub
node_download -->|"fetches files"| node_externalhub
node_builder -->|"builds dataset"| node_dataset
node_builder -.->|"builds stream"| node_iterable
node_dataset -->|"stores tables"| node_arrow
node_dataset -->|"uses schema"| node_features
node_dataset -->|"checks fingerprints"| node_cache
node_dataset -->|"runs operations"| node_transform
node_transform -.->|"dispatches workers"| node_parallel
node_dataset -->|"formats access"| node_formatting
node_formatting -->|"returns values"| node_frameworks
node_user -->|"uses similarity search"| node_search
node_search -->|"indexes examples"| node_dataset
node_user -->|"imports or exports"| node_io
node_user -.->|"accesses bucket data"| node_storage
node_user -->|"selects splits"| node_datasetdict

click node_loadapi "https://github.com/huggingface/datasets/blob/main/src/datasets/load.py"
click node_builder "https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py"
click node_formats "https://github.com/huggingface/datasets/tree/main/src/datasets/packaged_modules"
click node_download "https://github.com/huggingface/datasets/blob/main/src/datasets/download/download_manager.py"
click node_streamdownload "https://github.com/huggingface/datasets/blob/main/src/datasets/download/streaming_download_manager.py"
click node_hub "https://github.com/huggingface/datasets/blob/main/src/datasets/hub.py"
click node_dataset "https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py"
click node_iterable "https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py"
click node_datasetdict "https://github.com/huggingface/datasets/blob/main/src/datasets/dataset_dict.py"
click node_arrow "https://github.com/huggingface/datasets/blob/main/src/datasets/table.py"
click node_features "https://github.com/huggingface/datasets/blob/main/src/datasets/features/features.py"
click node_cache "https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py"
click node_transform "https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py"
click node_parallel "https://github.com/huggingface/datasets/blob/main/src/datasets/parallel/parallel.py"
click node_formatting "https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py"
click node_search "https://github.com/huggingface/datasets/blob/main/src/datasets/search.py"
click node_io "https://github.com/huggingface/datasets/tree/main/src/datasets/io"
click node_storage "https://github.com/huggingface/datasets/tree/main/src/datasets"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_loadapi,node_builder,node_formats,node_download,node_streamdownload,node_hub,node_user toneBlue
class node_dataset,node_iterable,node_datasetdict,node_arrow,node_features,node_cache toneAmber
class node_transform,node_parallel,node_formatting,node_search toneMint
class node_frameworks,node_io,node_storage toneRose
class node_externalhub toneIndigo
```

## Components

### Loading and sources

- [Load API](https://github.com/huggingface/datasets/blob/main/src/datasets/load.py): `src/datasets/load.py`
- [Builder lifecycle](https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py): `src/datasets/builder.py`
- [Packaged builders](https://github.com/huggingface/datasets/tree/main/src/datasets/packaged_modules): `src/datasets/packaged_modules`
- [Download manager](https://github.com/huggingface/datasets/blob/main/src/datasets/download/download_manager.py): `src/datasets/download/download_manager.py`
- [Streaming manager](https://github.com/huggingface/datasets/blob/main/src/datasets/download/streaming_download_manager.py): `src/datasets/download/streaming_download_manager.py`
- [Hub access](https://github.com/huggingface/datasets/blob/main/src/datasets/hub.py): `src/datasets/hub.py`

### Dataset core

- [Dataset API](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py): `src/datasets/arrow_dataset.py`
- [Iterable dataset](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py): `src/datasets/iterable_dataset.py`
- [Dataset splits](https://github.com/huggingface/datasets/blob/main/src/datasets/dataset_dict.py): `src/datasets/dataset_dict.py`
- [Arrow tables](https://github.com/huggingface/datasets/blob/main/src/datasets/table.py): `src/datasets/table.py`
- [Typed features](https://github.com/huggingface/datasets/blob/main/src/datasets/features/features.py): `src/datasets/features/features.py`
- [Cache fingerprints](https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py): `src/datasets/fingerprint.py`

### Processing and access

- [Dataset transforms](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py): `src/datasets/arrow_dataset.py`
- [Parallel processing](https://github.com/huggingface/datasets/blob/main/src/datasets/parallel/parallel.py): `src/datasets/parallel/parallel.py`
- [Output formatting](https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py): `src/datasets/formatting/formatting.py`
- [Search indexes](https://github.com/huggingface/datasets/blob/main/src/datasets/search.py): `src/datasets/search.py`

### Interoperability and sharing

- **ML frameworks**
- [Import/export](https://github.com/huggingface/datasets/tree/main/src/datasets/io): `src/datasets/io`
- [Storage buckets](https://github.com/huggingface/datasets/tree/main/src/datasets): `src/datasets`

### Other components

- **Library user**
- **Hugging Face Hub**

## Connections

- Library user → Load API: requests dataset
- Load API → Builder lifecycle: creates builder
- Builder lifecycle → Packaged builders: selects format
- Builder lifecycle → Download manager: downloads files
- Builder lifecycle → Streaming manager: prepares stream
- Load API → Hub access: resolves Hub data
- Hub access → Hugging Face Hub: queries repository
- Download manager → Hugging Face Hub: fetches files
- Builder lifecycle → Dataset API: builds dataset
- Builder lifecycle → Iterable dataset: builds stream
- Dataset API → Arrow tables: stores tables
- Dataset API → Typed features: uses schema
- Dataset API → Cache fingerprints: checks fingerprints
- Dataset API → Dataset transforms: runs operations
- Dataset transforms → Parallel processing: dispatches workers
- Dataset API → Output formatting: formats access
- Output formatting → ML frameworks: returns values
- Library user → Search indexes: uses similarity search
- Search indexes → Dataset API: indexes examples
- Library user → Import/export: imports or exports
- Library user → Storage buckets: accesses bucket data
- Library user → Dataset splits: selects splits
