# huggingface/xet-core architecture

> Architecture diagram of the GitHub repository huggingface/xet-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/xet-core
- Repository: https://github.com/huggingface/xet-core

## Overview

##### Project Type and Purpose
- xet-core is a Rust multi-crate library suite with Python bindings and a Rust-based CLI.
- Purpose: provide chunk-based deduplication, local caching, and remote CAS communication for huggingface_hub (Python) and git_xet (CLI) to store and retrieve large binary models/datasets efficiently.

##### High-Level Architectural Components
a. Client SDKs / CLIs
- Python layer: `hf_xet` crate → produces `hf-xet` Python wheel used by huggingface_hub
- Rust CLI: `git_xet` crate → Git LFS–compatible agent for users via `git xet` commands
b. Core Rust Libraries
- Deduplication Pipeline:
- data (main dedupe driver)
- deduplication (chunker, aggregator)
- mdb_shard (shard storage format & recovery)
- cas_object & merklehash (object formats & hashing)
- Network I/O:
- cas_client (HTTP transport to CAS backend)
- hub_client (authentication & HF Hub APIs)
- Local Caching & Utilities:
- chunk_cache (disk-based chunk store)
- file_utils, utils, progress_tracking, error_printer, xet_runtime
c. Remote Storage & Services
- HF Hub Xet backend (CAS / shard services)
- Authentication & token services

##### Relationships and Data Flows
- Upload Flow:
1. User calls `huggingface_hub.upload_files()` → Python code calls into `hf_xet`
2. `hf_xet` → `data::FilePointerTranslator` orchestrates: chunking (deduplication), local cache lookup, remote upload via `cas_client`
3. `cas_client` → HTTP calls to CAS backend → store shards / chunks
- Download Flow:
1. Python or `git_xet` CLI requests files → `hf_xet` / `git_xet` calls into dedupe pipeline
2. Missing chunks fetched from CAS → stored in `chunk_cache` → assembled & written to disk
- Git Integration:
- `git_xet` installs Git LFS hooks → intercepts pushes/pulls → routes data through xet-core pipeline
- Diagnostics & Logging:
- `hf-xet-diag-*.sh` scripts collect logs, core dumps via runtime hooks in `xet_runtime`, profiling modules, and `error_printer`.

##### Architectural Patterns & Design Principles
- Modular multi-crate architecture: crates isolated by responsibility (networking, caching, data processing).
- Singleflight & retry wrappers to dedupe in-flight requests (`utils::singleflight`, `cas_client::retry_wrapper`).
- Tokio async runtimes with optional console integration (`hf_xet` features).
- Python FFI via PyO3/maturin for seamless Python integration.
- Clear abstraction layers:
- Interface crates (`cas_types`, `deduplication::interface`, `data::remote_client_interface`)
- Implementation crates (cas_client, chunk_cache, mdb_shard)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Python and CLI Layers
    subgraph "Python Layer"
        HH["huggingface_hub (Python SDK)"]:::python
        HF["hf-xet Python Binding"]:::client
    end

    subgraph "Rust CLI Layer"
        GX["git_xet CLI"]:::client
    end

    %% Deduplication Pipeline
    subgraph "Deduplication Pipeline"
        DP["data (orchestrator)"]:::data
        CH["deduplication (chunker & aggregator)"]:::data
        CO["cas_object"]:::data
        MH["merklehash"]:::data
        MS["mdb_shard"]:::data
    end

    %% Caching & Networking
    subgraph "Caching & Networking"
        CC["chunk_cache (local disk)"]:::storage
        CASC["cas_client (HTTP)"]:::network
        HC["hub_client (HF API & auth)"]:::network
    end

    %% Utilities & Types
    subgraph "Utilities & Shared Types"
        FU["file_utils"]:::util
        UT["utils (singleflight, retries)"]:::util
        PT["progress_tracking"]:::util
        EP["error_printer"]:::util
        XR["xet_runtime"]:::util
        CT["cas_types"]:::types
    end

    %% Remote Backend
    R["HF Hub Xet CAS Backend"]:::remote

    %% Flows
    HH -->|FFI/PyO3| HF
    HF -->|calls orchestrator| DP
    GX -->|calls orchestrator| DP

    DP --> CH
    CH --> CO
    CO --> MH
    MH --> MS

    DP -->|local FS I/O| CC
    DP -->|async HTTP/Tokio| CASC
    HF -->|auth & HF API| HC

    CASC -->|HTTP| R
    HC -->|HTTP| R

    %% Click Events
    click HF "https://github.com/huggingface/xet-core/tree/main/hf_xet/"
    click GX "https://github.com/huggingface/xet-core/tree/main/git_xet/"
    click DP "https://github.com/huggingface/xet-core/tree/main/data/"
    click CH "https://github.com/huggingface/xet-core/tree/main/deduplication/"
    click MS "https://github.com/huggingface/xet-core/tree/main/mdb_shard/"
    click CO "https://github.com/huggingface/xet-core/tree/main/cas_object/"
    click MH "https://github.com/huggingface/xet-core/tree/main/merklehash/"
    click CASC "https://github.com/huggingface/xet-core/tree/main/cas_client/"
    click HC "https://github.com/huggingface/xet-core/tree/main/hub_client/"
    click CC "https://github.com/huggingface/xet-core/tree/main/chunk_cache/"
    click FU "https://github.com/huggingface/xet-core/tree/main/file_utils/"
    click UT "https://github.com/huggingface/xet-core/tree/main/utils/"
    click PT "https://github.com/huggingface/xet-core/tree/main/progress_tracking/"
    click EP "https://github.com/huggingface/xet-core/tree/main/error_printer/"
    click XR "https://github.com/huggingface/xet-core/tree/main/xet_runtime/"
    click CT "https://github.com/huggingface/xet-core/tree/main/cas_types/"

    %% Styles
    classDef python fill:#d8bfd8,stroke:#000,stroke-width:1px
    classDef client fill:#add8e6,stroke:#000,stroke-width:1px
    classDef data fill:#90ee90,stroke:#000,stroke-width:1px
    classDef network fill:#87cefa,stroke:#000,stroke-width:1px
    classDef storage fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef remote fill:#ffa500,stroke:#000,stroke-width:1px
    classDef util fill:#dda0dd,stroke:#000,stroke-width:1px
    classDef types fill:#ffb6c1,stroke:#000,stroke-width:1px
```
