How deepseek-ai/3fs works
Major System Components
• Clients & Frontend:
– This includes the FUSE layer (src/fuse) and CLI tools (found under src/client and hf3fs_utils). Clients interact with 3FS either via the virtual file system interface or through admin and benchmark tools.
• Metadata & Transactional Services:
– The project integrates a stateless metadata service that works over a transactional key-value store (FoundationDB as per README and indications in src/meta, src/core, and supporting libraries in src/common). This module holds filesystem metadata and supports chain replication (CRAQ).
• Management Daemon (mgmtd):
– Modules such as those found in src/mgmtd and subdirectories (including background, ops, service, and store) manage cluster configuration, node registration, and overall system health. They also deal with consistent configuration and replication.
• Data/Storage Layer:
– The storage subsystem (works in src/storage, src/storage/chunk_engine, and src/storage/service) manages actual data blocks, chunk allocation, reading/writing with RDMA/network optimizations, and implements advanced features like parallel checkpointing and KVCache.
• Supporting Services & Libraries:
– There are libraries and common utilities (src/common, src/lib, src/kv) handling networking (RDMA, TCP, etc.), logging, error handling, serialization (src/common/serde), and memory management (src/memory) which both support and interlink the main modules.
• Benchmarking/Testing:
– The benchmarks and tests directories (e.g., benchmarks/fio_usrbio and various tests directories) validate performance and help simulate workloads. Although not part of the runtime architecture, they provide insight into the system’s expected performance and resilience.
Relationships and Data Flows
• Client requests (file reads/writes, administrative commands) enter via the FUSE interface or CLI.
• These requests are routed to the metadata service, where the stateless metadata server uses FoundationDB as its backend.
• The management daemon (mgmtd) then coordinates node registration, configuration updates, and chain replication processes (using CRAQ algorithms) across storage nodes.
• Storage nodes, which often run the underlying chunk engine and storage service components, perform data operations using RDMA and asynchronous I/O.
• Logging, monitoring, and performance metrics (as seen in src/common/monitor, among others) continuously feed back status information to mgmtd and possibly to client dashboards.
• There is a clear separation between the control/metadata plane (mgmtd, meta) and the data plane (storage service and chunk engine).
Architectural Patterns and Design Principles
• Disaggregated Architecture: Separates metadata, management, and storage roles.
• Chain Replication with CRAQ: Ensures strong consistency amid distributed storage nodes.
• Modular/Layered Design: Notice the separation into client, core, metadata, management, storage, and utility libraries. This not only supports clean modularity but also allows components to be developed, tested, and maintained independently.
• Extensibility: With support for multiple interfaces (FUSE, CLI, APIs) and plug-in libraries (e.g., the Rust-based USRBIO component in src/lib/rs), the design emphasizes extensibility.
Relevant Technologies and Frameworks
• C++ and Rust (notice Cargo.toml files in the Rust components under src/lib/rs and elsewhere).
• FoundationDB as the transactional key-value store.
• RDMA and InfiniBand for high bandwidth/low latency transfers.
• Libfuse for FUSE-based file system integration.
• Standard build/deployment tools: CMake, Docker (dockerfiles), and related CI/CD workflows defined in .github/workflows.