# deepseek-ai/3fs architecture

> Architecture diagram of the GitHub repository deepseek-ai/3fs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepseek-ai/3fs
- Repository: https://github.com/deepseek-ai/3fs

## Overview

##### Major System Components
- Clients & Frontend:
– This includes the FUSE layer (src/fuse) and CLI tools (found under src/client and hf3fs_utils). Clients interact with 3FS either via the virtual file system interface or through admin and benchmark tools.
- Metadata & Transactional Services:
– The project integrates a stateless metadata service that works over a transactional key-value store (FoundationDB as per README and indications in src/meta, src/core, and supporting libraries in src/common). This module holds filesystem metadata and supports chain replication (CRAQ).
- Management Daemon (mgmtd):
– Modules such as those found in src/mgmtd and subdirectories (including background, ops, service, and store) manage cluster configuration, node registration, and overall system health. They also deal with consistent configuration and replication.
- Data/Storage Layer:
– The storage subsystem (works in src/storage, src/storage/chunk_engine, and src/storage/service) manages actual data blocks, chunk allocation, reading/writing with RDMA/network optimizations, and implements advanced features like parallel checkpointing and KVCache.
- Supporting Services & Libraries:
– There are libraries and common utilities (src/common, src/lib, src/kv) handling networking (RDMA, TCP, etc.), logging, error handling, serialization (src/common/serde), and memory management (src/memory) which both support and interlink the main modules.
- Benchmarking/Testing:
– The benchmarks and tests directories (e.g., benchmarks/fio_usrbio and various tests directories) validate performance and help simulate workloads. Although not part of the runtime architecture, they provide insight into the system’s expected performance and resilience.

##### Relationships and Data Flows
- Client requests (file reads/writes, administrative commands) enter via the FUSE interface or CLI.
- These requests are routed to the metadata service, where the stateless metadata server uses FoundationDB as its backend.
- The management daemon (mgmtd) then coordinates node registration, configuration updates, and chain replication processes (using CRAQ algorithms) across storage nodes.
- Storage nodes, which often run the underlying chunk engine and storage service components, perform data operations using RDMA and asynchronous I/O.
- Logging, monitoring, and performance metrics (as seen in src/common/monitor, among others) continuously feed back status information to mgmtd and possibly to client dashboards.
- There is a clear separation between the control/metadata plane (mgmtd, meta) and the data plane (storage service and chunk engine).

##### Architectural Patterns and Design Principles
- Disaggregated Architecture: Separates metadata, management, and storage roles.
- Chain Replication with CRAQ: Ensures strong consistency amid distributed storage nodes.
- Modular/Layered Design: Notice the separation into client, core, metadata, management, storage, and utility libraries. This not only supports clean modularity but also allows components to be developed, tested, and maintained independently.
- Extensibility: With support for multiple interfaces (FUSE, CLI, APIs) and plug-in libraries (e.g., the Rust-based USRBIO component in src/lib/rs), the design emphasizes extensibility.

##### Relevant Technologies and Frameworks
- C++ and Rust (notice Cargo.toml files in the Rust components under src/lib/rs and elsewhere).
- FoundationDB as the transactional key-value store.
- RDMA and InfiniBand for high bandwidth/low latency transfers.
- Libfuse for FUSE-based file system integration.
- Standard build/deployment tools: CMake, Docker (dockerfiles), and related CI/CD workflows defined in .github/workflows.

## Diagram (Mermaid)

```mermaid
flowchart TD

    subgraph "Clients & Frontend"
        FUSE["FUSE Layer"]:::clients
        CLI1["CLI Tools (src/client)"]:::clients
        CLI2["CLI Tools (hf3fs_utils)"]:::clients
    end

    subgraph "Metadata & Transactional Services"
        META["Metadata Service"]:::metadata
        KV["Transactional KV Store"]:::metadata
    end

    subgraph "Management Daemon"
        MGMT["Management Daemon (mgmtd)"]:::management
    end

    subgraph "Data/Storage Layer"
        STORAGE["Storage Service"]:::storage
        CHUNK["Chunk Engine"]:::storage
    end

    subgraph "Supporting Services & Libraries"
        UTIL["Common Utilities"]:::support
        LIB["Lib API"]:::support
        NET["Networking"]:::support
        MEM["Memory Management"]:::support
    end

    subgraph "Benchmarking & Testing"
        BENCH["Benchmarks"]:::benchmark
        TEST["Tests"]:::benchmark
    end

    %% Connections between major components
    FUSE -->|"fileIO"| META
    CLI1 -->|"admin"| MGMT
    CLI2 -->|"adminCLI"| MGMT
    CLI2 -->|"query"| META

    META -->|"persist"| KV

    MGMT -->|"chainRep"| STORAGE
    MGMT -->|"chainRep"| CHUNK

    STORAGE -->|"dataReturn"| FUSE
    CHUNK -->|"dataReturn"| FUSE

    STORAGE -->|"monitor"| MGMT
    CHUNK -->|"monitor"| MGMT

    %% Supporting services usage arrows
    META -->|"uses"| LIB
    MGMT -->|"uses"| NET
    STORAGE -->|"uses"| MEM
    UTIL -->|"logs"| MGMT

    %% Click Events
    click FUSE "https://github.com/deepseek-ai/3fs/blob/main/src/fuse/FuseApplication.cc"
    click CLI1 "https://github.com/deepseek-ai/3fs/tree/main/src/client/cli"
    click CLI2 "https://github.com/deepseek-ai/3fs/blob/main/hf3fs_utils/cli.py"
    click META "https://github.com/deepseek-ai/3fs/blob/main/src/meta/service/MetaServer.cc"
    click KV "https://github.com/deepseek-ai/3fs/blob/main/src/common/kv/IKVEngine.h"
    click MGMT "https://github.com/deepseek-ai/3fs/blob/main/src/mgmtd/MgmtdServer.cc"
    click STORAGE "https://github.com/deepseek-ai/3fs/blob/main/src/storage/service/StorageService.cc"
    click CHUNK "https://github.com/deepseek-ai/3fs/blob/main/src/storage/chunk_engine/src/lib.rs"
    click UTIL "https://github.com/deepseek-ai/3fs/blob/main/src/common/utils/StringUtils.cc"
    click LIB "https://github.com/deepseek-ai/3fs/blob/main/src/lib/api/hf3fs.h"
    click NET "https://github.com/deepseek-ai/3fs/blob/main/src/common/net/Transport.cc"
    click MEM "https://github.com/deepseek-ai/3fs/blob/main/src/memory/common/GlobalMemoryAllocator.cc"
    click BENCH "https://github.com/deepseek-ai/3fs/blob/main/benchmarks/fio_usrbio/hf3fs_usrbio.cpp"
    click TEST "https://github.com/deepseek-ai/3fs/tree/main/tests/fuse"

    %% Styles
    classDef clients fill:#b3cde0,stroke:#1f78b4,stroke-width:2px;
    classDef metadata fill:#ccebc5,stroke:#33a02c,stroke-width:2px;
    classDef management fill:#ffed6f,stroke:#ff7f00,stroke-width:2px;
    classDef storage fill:#fbb4ae,stroke:#e31a1c,stroke-width:2px;
    classDef support fill:#decbe4,stroke:#6a3d9a,stroke-width:2px;
    classDef benchmark fill:#fddaec,stroke:#dd1c77,stroke-width:2px;
```
