How huggingface/xet-core works
Project Type and Purpose
• xet-core is a Rust multi-crate library suite with Python bindings and a Rust-based CLI.
• Purpose: provide chunk-based deduplication, local caching, and remote CAS communication for huggingface_hub (Python) and git_xet (CLI) to store and retrieve large binary models/datasets efficiently.
High-Level Architectural Components
a. Client SDKs / CLIs
• Python layer: hf_xet crate → produces hf-xet Python wheel used by huggingface_hub
• Rust CLI: git_xet crate → Git LFS–compatible agent for users via git xet commands
b. Core Rust Libraries
• Deduplication Pipeline:
• data (main dedupe driver)
• deduplication (chunker, aggregator)
• mdb_shard (shard storage format & recovery)
• cas_object & merklehash (object formats & hashing)
• Network I/O:
• cas_client (HTTP transport to CAS backend)
• hub_client (authentication & HF Hub APIs)
• Local Caching & Utilities:
• chunk_cache (disk-based chunk store)
• file_utils, utils, progress_tracking, error_printer, xet_runtime
c. Remote Storage & Services
• HF Hub Xet backend (CAS / shard services)
• Authentication & token services
Relationships and Data Flows
• Upload Flow:
1. User calls huggingface_hub.upload_files() → Python code calls into hf_xet
2. hf_xet → data::FilePointerTranslator orchestrates: chunking (deduplication), local cache lookup, remote upload via cas_client
3. cas_client → HTTP calls to CAS backend → store shards / chunks
• Download Flow:
1. Python or git_xet CLI requests files → hf_xet / git_xet calls into dedupe pipeline
2. Missing chunks fetched from CAS → stored in chunk_cache → assembled & written to disk
• Git Integration:
• git_xet installs Git LFS hooks → intercepts pushes/pulls → routes data through xet-core pipeline
• Diagnostics & Logging:
• hf-xet-diag-*.sh scripts collect logs, core dumps via runtime hooks in xet_runtime, profiling modules, and error_printer.
Architectural Patterns & Design Principles
• Modular multi-crate architecture: crates isolated by responsibility (networking, caching, data processing).
• Singleflight & retry wrappers to dedupe in-flight requests (utils::singleflight, cas_client::retry_wrapper).
• Tokio async runtimes with optional console integration (hf_xet features).
• Python FFI via PyO3/maturin for seamless Python integration.
• Clear abstraction layers:
• Interface crates (cas_types, deduplication::interface, data::remote_client_interface)
• Implementation crates (cas_client, chunk_cache, mdb_shard)