# momseul/tet architecture

> Architecture diagram of the GitHub repository momseul/tet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/momseul/tet
- Repository: https://github.com/momseul/tet

## Overview

##### Project Type & Purpose
- The project is a Distributed File System (DFS) inspired by Google File System (GFS) and Hadoop File System (HDFS).
- Its main purpose is to allow users to upload files that are split into chunks (or blocks), store these blocks on DataNodes, manage metadata and block replication through a NameNode, and finally enable file retrieval with reassembly by the Client.

##### Analyzing the File Structure
- The top-level directories include:
- client: Contains modules like main.py and chunks.py, which handle file partitioning, uploading, downloading, and reassembling.
- namenode: Holds modules (namenode.py, metadata.py, distributor.py) that manage file metadata, coordinate data block replication, and assign storage locations.
- datanode: Includes datanode.py, main.py, and storage.py, which are responsible for storing the file chunks and handling data requests.
- protos: Contains .proto files and their gRPC-generated Python stubs (both synchronous and type-hinted versions) which define the gRPC service interfaces between components.
- Additional files such as requirements.txt, LICENSE, and README provide context and dependencies.

##### Key Components
- Client:
- Acts as the user interface.
- Handles file chunking and reassembly.
- Initiates requests (upload/download) to the DFS.
- NameNode:
- Manages the centralized metadata.
- Coordinates the replication and distribution of file chunks.
- Acts as the system’s “brain” controlling how DataNodes store and manage blocks.
- DataNodes:
- Store the actual file chunks.
- Interact with the NameNode for receiving instructions on which blocks to store or replicate.
- gRPC API/Communication layer:
- Both NameNode and DataNodes communicate through the defined gRPC services (as provided in the protos folder).
- This layer should be depicted as the mechanism facilitating inter-component communication.

##### Architectural Relationships & Data Flow
- Client → NameNode:
- When a client initiates a file upload, it first contacts the NameNode (using a gRPC call) to establish metadata records, retrieve the list of available DataNodes, and get instructions for block replication.
- Client ↔ DataNodes:
- The file is split into chunks by the client.
- The client uploads these chunks directly to the designated DataNodes.
- Similarly, for downloads, the client retrieves blocks from the DataNodes and reassembles them.
- NameNode ↔ DataNodes:
- The NameNode controls replication. It informs DataNodes where to store replicas of data blocks.
- DataNodes might periodically report status or heartbeat signals back to the NameNode.

##### Architectural Patterns & Design Principles
- Separation of Concerns:
- Different roles for each component (e.g., client for file handling, NameNode for metadata, DataNodes for storage).
- Distributed Communication:
- Utilizes gRPC for a clear, well-defined interface and message-passing between nodes.
- High Availability & Fault Tolerance:
- Block replication across multiple DataNodes ensures fault tolerance.
- Modularity:
- Each component can be developed, deployed, and even scaled independently.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Client Layer"
        C["Client (File Operations)"]
    end

    subgraph "gRPC Communication"
        G["gRPC API Layer"]
    end

    subgraph "NameNode Layer"
        N["NameNode (Metadata & Replication)"]
    end

    subgraph "DataNode Layer"
        D["DataNodes (Chunk Storage)"]
    end

    C -->|"RPC:metadata_request"| G
    C -->|"RPC:chunk_transfer"| G
    G -->|"RPC:route_to_NameNode"| N
    G -->|"RPC:route_to_DataNodes"| D
    N -->|"RPC:replication_instruct"| G
    G -->|"RPC:route_to_DataNodes"| D
    D -->|"RPC:heartbeat_status"| G
    G -->|"RPC:route_to_NameNode"| N

    classDef client fill:#FFA07A,stroke:#333,stroke-width:2px;
    classDef namenode fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef datanode fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef grpc fill:#DDA0DD,stroke:#333,stroke-width:2px;

    class C client;
    class N namenode;
    class D datanode;
    class G grpc;

    click C "https://github.com/momseul/tet/blob/main/client/main.py"
    click C "https://github.com/momseul/tet/blob/main/client/chunks.py"
    click N "https://github.com/momseul/tet/blob/main/namenode/namenode.py"
    click N "https://github.com/momseul/tet/blob/main/namenode/metadata.py"
    click N "https://github.com/momseul/tet/blob/main/namenode/distributor.py"
    click D "https://github.com/momseul/tet/blob/main/datanode/datanode.py"
    click D "https://github.com/momseul/tet/blob/main/datanode/main.py"
    click D "https://github.com/momseul/tet/blob/main/datanode/storage.py"
    click G "https://github.com/momseul/tet/tree/main/protos/"
```
