How aarrasseayoub01/hdfs-like works
Project Type and Purpose
• This is a distributed file‐system prototype (akin to HDFS) implemented as a set of microservices and a CLI tool.
• Core components:
• A Rust‐based CLI (hdfs_cli/) for end‐user commands.
• A NameNode service (hdfs_namenode/) in Go, which manages namespace metadata and coordinates clients and DataNodes.
• A DataNode service (hdfs_datanode/) in Go, which stores and serves file blocks.
• A shared protobuf definition (protobuf/) that defines the gRPC interfaces and message formats.
File Structure Analysis
• Top‐level directories:
• hdfs_cli/ (Rust CLI client)
• hdfs_namenode/ (Go service exposing gRPC server endpoints for metadata operations)
• hdfs_datanode/ (Go service exposing gRPC server endpoints for block operations)
• protobuf/ (Protocol Buffers definitions compiled for Go)
• Each Go service follows a layered pattern:
• internal/config for configuration
• internal/controller or internal/service for business logic
• internal/gRPC for server/client glue code
• Persistence or data‐management subpackages (e.g., persistence, fs, datamngnt)
• Makefile, go.mod, go.sum coordinate builds; Cargo.toml and Cargo.lock coordinate the Rust CLI.
README Insights
• Minimal README, but directory names and file contents make it clear: NameNode and DataNode communicate over gRPC using the protobuf definitions, the CLI invokes NameNode operations via gRPC, NameNode directs DataNode for block storage.
Detailed Architectural Elements to Capture
• The gRPC client modules inside each service (internal/gRPC/client.go) and server modules (internal/gRPC/server.go).
• The persistence subsystems in NameNode (internal/persistence) handling editlog and fsImage.
• The DataNode’s block manager (internal/datamngnt) and metadata operations.
• The CLI’s command‐parsing and HTTP/gRPC client (cli_commands.rs, http_client.rs).