How huggingface/datasets works
π€ Datasets loads public Hub datasets or local files, optionally streams them, builds typed Arrow-backed datasets, and exposes them for preprocessing and model use. The core path runs from the user through load_dataset and a format-specific builder, with download/streaming support and Arrow storage; Dataset operations then transform data, and formatters decode it for consumers. Search indexes, Hub publishing, and the broad set of packaged formats and multimodal features are additional capabilities. The graph reflects documented capabilities where implementation excerpts were unavailable; those links are omitted when call ownership is uncertain.
Main entry points: arrow_dataset.py Dataset API Β Β·Β builder.py Builder lifecycle Β Β·Β load.py Load API Β Β·Β download_manager.py Download manager
Components 21
Loading and sources
- Load API
src/datasets/load.py - Builder lifecycle
src/datasets/builder.py - Packaged builders
src/datasets/packaged_modules - Download manager
src/datasets/download/download_manager.py - Streaming manager
src/datasets/download/streaming_download_manager.py - Hub access
src/datasets/hub.py
Dataset core
- Dataset API
src/datasets/arrow_dataset.py - Iterable dataset
src/datasets/iterable_dataset.py - Dataset splits
src/datasets/dataset_dict.py - Arrow tables
src/datasets/table.py - Typed features
src/datasets/features/features.py - Cache fingerprints
src/datasets/fingerprint.py
Processing and access
- Dataset transforms
src/datasets/arrow_dataset.py - Parallel processing
src/datasets/parallel/parallel.py - Output formatting
src/datasets/formatting/formatting.py - Search indexes
src/datasets/search.py
Interoperability and sharing
- ML frameworks
- Import/export
src/datasets/io - Storage buckets
src/datasets
Other components
- Library user
- Hugging Face Hub
Connections 22
- Library user β Load API: requests dataset
- Load API β Builder lifecycle: creates builder
- Builder lifecycle β Packaged builders: selects format
- Builder lifecycle β Download manager: downloads files
- Builder lifecycle β Streaming manager: prepares stream
- Load API β Hub access: resolves Hub data
- Hub access β Hugging Face Hub: queries repository
- Download manager β Hugging Face Hub: fetches files
- Builder lifecycle β Dataset API: builds dataset
- Builder lifecycle β Iterable dataset: builds stream
- Dataset API β Arrow tables: stores tables
- Dataset API β Typed features: uses schema
- Dataset API β Cache fingerprints: checks fingerprints
- Dataset API β Dataset transforms: runs operations
- Dataset transforms β Parallel processing: dispatches workers
- Dataset API β Output formatting: formats access
- Output formatting β ML frameworks: returns values
- Library user β Search indexes: uses similarity search
- Search indexes β Dataset API: indexes examples
- Library user β Import/export: imports or exports
- Library user β Storage buckets: accesses bucket data
- Library user β Dataset splits: selects splits