# aarrasseayoub01/hdfs-like architecture

> Architecture diagram of the GitHub repository aarrasseayoub01/hdfs-like, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aarrasseayoub01/hdfs-like
- Repository: https://github.com/aarrasseayoub01/hdfs-like

## Overview

##### Project Type and Purpose
- This is a distributed file‐system prototype (akin to HDFS) implemented as a set of microservices and a CLI tool.
- Core components:
- A Rust‐based CLI (`hdfs_cli/`) for end‐user commands.
- A NameNode service (`hdfs_namenode/`) in Go, which manages namespace metadata and coordinates clients and DataNodes.
- A DataNode service (`hdfs_datanode/`) in Go, which stores and serves file blocks.
- A shared protobuf definition (`protobuf/`) that defines the gRPC interfaces and message formats.

##### File Structure Analysis
- Top‐level directories:
- `hdfs_cli/` (Rust CLI client)
- `hdfs_namenode/` (Go service exposing gRPC server endpoints for metadata operations)
- `hdfs_datanode/` (Go service exposing gRPC server endpoints for block operations)
- `protobuf/` (Protocol Buffers definitions compiled for Go)
- Each Go service follows a layered pattern:
- `internal/config` for configuration
- `internal/controller` or `internal/service` for business logic
- `internal/gRPC` for server/client glue code
- Persistence or data‐management subpackages (e.g., `persistence`, `fs`, `datamngnt`)
- `Makefile`, `go.mod`, `go.sum` coordinate builds; `Cargo.toml` and `Cargo.lock` coordinate the Rust CLI.

##### README Insights
- Minimal README, but directory names and file contents make it clear: NameNode and DataNode communicate over gRPC using the protobuf definitions, the CLI invokes NameNode operations via gRPC, NameNode directs DataNode for block storage.

##### Detailed Architectural Elements to Capture
- The gRPC client modules inside each service (`internal/gRPC/client.go`) and server modules (`internal/gRPC/server.go`).
- The persistence subsystems in NameNode (`internal/persistence`) handling `editlog` and `fsImage`.
- The DataNode’s block manager (`internal/datamngnt`) and metadata operations.
- The CLI’s command‐parsing and HTTP/gRPC client (`cli_commands.rs`, `http_client.rs`).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Client
    subgraph "CLI Client (Rust)"
        CLICmd["main.rs"]:::cli
        CLICommands["cli_commands.rs"]:::cli
        CLIHttp["http_client.rs"]:::cli
        CLImodels["models.rs"]:::cli
        CLIutils["utils.rs"]:::cli
    end

    %% Shared Protobuf Contract
    subgraph "Shared Protobuf/gRPC Contract"
        ProtoDef["hdfs.proto"]:::contract
        ProtoGo["hdfs.pb.go"]:::contract
        ProtoGrpc["hdfs_grpc.pb.go"]:::contract
    end

    %% NameNode Service
    subgraph "NameNode Service (Go)"
        NNMain["main.go"]:::service
        NNgRPCServer["server.go"]:::grpc
        NNgRPCClient["client.go"]:::grpc
        NNController["controller.go"]:::business
        NNServiceLogic["service.go"]:::business
        NNMeta["metadata.go"]:::business
        NNTools["tools.go"]:::business
        NNEditLog["editlog.go"]:::persistence
        NNFsImage["fsImage.go"]:::persistence
        NNCheckpoint["checkpoint.go"]:::persistence
        NBBoot["boot.go"]:::persistence
    end

    %% DataNode Service
    subgraph "DataNode Service (Go)"
        DNMain["main.go"]:::service
        DNgRPCServer["server.go"]:::grpc
        DNgRPCClient["client.go"]:::grpc
        DNController["controller.go"]:::business
        DNLogic["datanode.go"]:::business
        DNDataMgr["data_manger.go"]:::persistence
        DNMeta["metadata.go"]:::persistence
        DNUtil["util.go"]:::persistence
        DNConfig["config.go"]:::service
    end

    %% Persistent Storage
    StorageNN["Metadata Storage"]:::storage
    StorageDN["Block Storage"]:::storage

    %% Interactions
    CLIHttp -->|"gRPC Command"| NNgRPCServer
    NNgRPCServer -->|"uses"| ProtoDef
    NNgRPCServer -->|"uses"| ProtoGo
    NNgRPCServer -->|"uses"| ProtoGrpc
    NNgRPCClient -->|"uses"| ProtoDef
    NNgRPCClient -->|"uses"| ProtoGo
    NNgRPCClient -->|"uses"| ProtoGrpc

    DNgRPCServer -->|"uses"| ProtoDef
    DNgRPCServer -->|"uses"| ProtoGo
    DNgRPCServer -->|"uses"| ProtoGrpc
    DNgRPCClient -->|"uses"| ProtoDef
    DNgRPCClient -->|"uses"| ProtoGo
    DNgRPCClient -->|"uses"| ProtoGrpc

    NNgRPCClient -->|"block report/heartbeat"| DNgRPCServer
    DNgRPCServer -->|"instructions"| NNgRPCClient

    NNEditLog -->|"write metadata"| StorageNN
    NNFsImage -->|"snapshot metadata"| StorageNN
    NNCheckpoint -->|"checkpoint"| StorageNN

    DNDataMgr -->|"store/retrieve blocks"| StorageDN

    %% Click Events CLI
    click CLICmd "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_cli/src/main.rs"
    click CLICommands "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_cli/src/cli_commands.rs"
    click CLIHttp "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_cli/src/http_client.rs"
    click CLImodels "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_cli/src/models.rs"
    click CLIutils "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_cli/src/utils.rs"

    %% Click Events Protobuf
    click ProtoDef "https://github.com/aarrasseayoub01/hdfs-like/blob/main/protobuf/hdfs.proto"
    click ProtoGo "https://github.com/aarrasseayoub01/hdfs-like/blob/main/protobuf/hdfs.pb.go"
    click ProtoGrpc "https://github.com/aarrasseayoub01/hdfs-like/blob/main/protobuf/hdfs_grpc.pb.go"

    %% Click Events NameNode
    click NNMain "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/cmd/server/main.go"
    click NNgRPCServer "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/gRPC/server.go"
    click NNgRPCClient "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/gRPC/client.go"
    click NNController "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/controller/controller.go"
    click NNServiceLogic "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/service/service.go"
    click NNMeta "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/fs/metadata.go"
    click NNTools "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/fs/tools.go"
    click NNEditLog "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/persistence/editlog.go"
    click NNFsImage "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/persistence/fsImage.go"
    click NNCheckpoint "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/persistence/checkpoint.go"
    click NBBoot "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_namenode/internal/persistence/boot.go"

    %% Click Events DataNode
    click DNMain "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/cmd/datanode/main.go"
    click DNgRPCServer "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/gRPC/server.go"
    click DNgRPCClient "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/gRPC/client.go"
    click DNController "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/controller/controller.go"
    click DNLogic "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/datanode/datanode.go"
    click DNDataMgr "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/datamngnt/data_manger.go"
    click DNMeta "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/datamngnt/metadata.go"
    click DNUtil "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/datamngnt/util.go"
    click DNConfig "https://github.com/aarrasseayoub01/hdfs-like/blob/main/hdfs_datanode/internal/config/config.go"

    %% Styles
    classDef cli fill:#B3D4FC,stroke:#2B6CB0,color:#000
    classDef service fill:#C6F6D5,stroke:#2F855A,color:#000
    classDef grpc fill:#FEEBC8,stroke:#DD6B20,color:#000
    classDef business fill:#E9D8FD,stroke:#6B46C1,color:#000
    classDef persistence fill:#EDF2F7,stroke:#A0AEC0,color:#000
    classDef contract fill:#F0FFF4,stroke:#38A169,color:#000
    classDef storage fill:#F7FAFC,stroke:#718096,color:#000
```
