How juicedata/juicefs works
Purpose and executable entry points
JuiceFS is a Go-based, cloud-native distributed POSIX filesystem client. It separates filesystem metadata from file data: metadata resides in a transactional database and chunked data in object or filesystem-backed storage. The root executable delegates to the CLI command layer: main.go, cmd/main.go, and command implementations such as cmd/mount.go, cmd/format.go, and cmd/gateway.go.
Client composition and mount lifecycle
The mount command constructs the storage stack, virtual filesystem, and OS-specific mount adapter. pkg/vfs implements filesystem semantics over metadata and chunk storage; pkg/fuse exposes it to Linux/macOS through FUSE, while pkg/winfsp supplies the Windows integration. Primary paths: cmd/mount.go, pkg/vfs/vfs.go, pkg/fuse/fuse.go, pkg/winfsp/winfs.go.
Metadata control plane
pkg/meta defines the metadata abstraction and filesystem state: namespaces, inode/file attributes, directory entries, slice layouts, locks, quotas, status, backups, and dump/load. Concrete backends span Redis, SQL databases, and transactional KV engines, making this package the consistency and coordination boundary between clients. Paths: pkg/meta/interface.go, pkg/meta/redis.go, pkg/meta/sql.go, pkg/meta/tkv.go.
Data plane: file layout, caching, and object persistence
File writes are represented as chunk/slice metadata and materialized as fixed-size blocks in the configured object store. pkg/chunk coordinates block reads/writes, memory/disk caches, prefetching, eviction, and deduplication of concurrent fetches. The VFS binds metadata operations to this chunk store. Paths: pkg/vfs/writer.go, pkg/vfs/reader.go, pkg/chunk/cached_store.go, pkg/chunk/cache_manager.go.
Storage-provider boundary
pkg/object provides the common object-storage interface plus adapters for S3-compatible services, public clouds, HDFS, local filesystems, Redis, SQL, and network filesystems. Wrappers add encryption, prefixes, sharding, checksums, and REST behavior without coupling VFS/chunk logic to a provider. Paths: pkg/object/interface.go, pkg/object/object_storage.go, pkg/object/s3.go, pkg/object/encrypt.go.
Alternative access interfaces and integrations
Beyond POSIX mounts, the client offers S3 gateway and WebDAV services, plus a Hadoop-compatible Java filesystem SDK. The Java implementation bridges to Go-native filesystem functionality and integrates with Hadoop/Flink ecosystems; Python packaging exposes another client-facing SDK surface. Paths: pkg/gateway/gateway.go, cmd/webdav.go, sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java, sdk/python/juicefs/juicefs/juicefs.py.
Operations, observability, and deployment boundaries
CLI operational commands format/destroy filesystems, inspect or repair metadata, manage garbage collection, compact data, clone/restore, synchronize, and report status. Metrics are emitted from filesystem/chunk layers; optional anonymous usage reporting is isolated. Runtime targets include Linux, macOS, and Windows; deployments commonly connect to external metadata engines (Redis/MySQL/PostgreSQL/SQLite/TiKV/etcd/FoundationDB) and object stores, with Kubernetes integration documented/deployed separately. Paths: cmd/fsck.go, cmd/gc.go, cmd/sync.go, pkg/metric/metrics.go, pkg/usage/usage.go.
Main entry points: vfs.go Virtual filesystem · cached_store.go Chunk store · mount.go Mount command · main.go CLI commands
Components 21
Commands
Executable and operational command layer.
- Root executable
main.go - CLI commands
cmd/main.go - Mount command
cmd/mount.go - Operations commands · Repair, garbage collection, synchronization, and filesystem administration.
cmd/fsck.go
Filesystem Core
Metadata, VFS, and chunk data-path layers.
- Virtual filesystem · Binds metadata state to file-data operations.
pkg/vfs/vfs.go - VFS readers and writers · Translates file reads and writes into chunk and slice operations.
pkg/vfs/writer.go - Metadata abstraction · Transactional namespace, inode, slice, lock, quota, and lifecycle state.
pkg/meta/interface.go - Chunk store · Coordinates fixed-size block reads, writes, caching, prefetch, and deduplicated fetches.
pkg/chunk/cached_store.go - Cache manager
pkg/chunk/cache_manager.go - Metrics
pkg/metric/metrics.go - Usage reporting · Isolated anonymous usage reporting.
pkg/usage/usage.go
Access Interfaces
Mount and protocol-facing adapters.
- FUSE adapter
pkg/fuse/fuse.go - WinFSP adapter
pkg/winfsp/winfs.go - S3 gateway
pkg/gateway/gateway.go - WebDAV command
cmd/webdav.go - Hadoop Java SDK · Bridges Go-native filesystem capabilities to Hadoop and Flink.
sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java - Python SDK
sdk/python/juicefs/juicefs/juicefs.py
Storage Providers
Provider-neutral persistence boundaries.
- Metadata backends
pkg/meta/sql.go - Object interface and adapters · Uniform object API with cloud, filesystem, database, and network adapters.
pkg/object/interface.go - Object wrappers · Applies encryption, prefixes, sharding, checksums, and REST behavior.
pkg/object/encrypt.go - External object stores
Connections 25
- Root executable → CLI commands: delegates to
- CLI commands → Mount command: runs
- CLI commands → Operations commands: runs
- Mount command → Virtual filesystem: constructs
- Mount command → FUSE adapter: mounts on Linux/macOS
- Mount command → WinFSP adapter: mounts on Windows
- FUSE adapter → Virtual filesystem: exposes
- WinFSP adapter → Virtual filesystem: exposes
- Virtual filesystem → Metadata abstraction: stores filesystem state
- Virtual filesystem → VFS readers and writers: performs file I/O
- VFS readers and writers → Chunk store: reads and writes blocks
- Metadata abstraction → Metadata backends: implemented by
- Chunk store → Cache manager: manages
- Chunk store → Object interface and adapters: persists blocks via
- Object interface and adapters → Object wrappers: composed with
- Object wrappers → External object stores: targets
- S3 gateway → Virtual filesystem: serves filesystem through
- WebDAV command → Virtual filesystem: serves filesystem through
- Hadoop Java SDK → Virtual filesystem: bridges to
- Python SDK → Virtual filesystem: client surface for
- Virtual filesystem → Metrics: emits
- Chunk store → Metrics: emits
- Virtual filesystem → Usage reporting: optionally reports
- Operations commands → Metadata abstraction: inspects and repairs
- Operations commands → Chunk store: GC, compacts, and syncs