How momseul/tet works
Project Type & Purpose
• The project is a Distributed File System (DFS) inspired by Google File System (GFS) and Hadoop File System (HDFS).
• Its main purpose is to allow users to upload files that are split into chunks (or blocks), store these blocks on DataNodes, manage metadata and block replication through a NameNode, and finally enable file retrieval with reassembly by the Client.
Analyzing the File Structure
• The top-level directories include:
• client: Contains modules like main.py and chunks.py, which handle file partitioning, uploading, downloading, and reassembling.
• namenode: Holds modules (namenode.py, metadata.py, distributor.py) that manage file metadata, coordinate data block replication, and assign storage locations.
• datanode: Includes datanode.py, main.py, and storage.py, which are responsible for storing the file chunks and handling data requests.
• protos: Contains .proto files and their gRPC-generated Python stubs (both synchronous and type-hinted versions) which define the gRPC service interfaces between components.
• Additional files such as requirements.txt, LICENSE, and README provide context and dependencies.
Key Components
• Client:
• Acts as the user interface.
• Handles file chunking and reassembly.
• Initiates requests (upload/download) to the DFS.
• NameNode:
• Manages the centralized metadata.
• Coordinates the replication and distribution of file chunks.
• Acts as the system’s “brain” controlling how DataNodes store and manage blocks.
• DataNodes:
• Store the actual file chunks.
• Interact with the NameNode for receiving instructions on which blocks to store or replicate.
• gRPC API/Communication layer:
• Both NameNode and DataNodes communicate through the defined gRPC services (as provided in the protos folder).
• This layer should be depicted as the mechanism facilitating inter-component communication.
Architectural Relationships & Data Flow
• Client → NameNode:
• When a client initiates a file upload, it first contacts the NameNode (using a gRPC call) to establish metadata records, retrieve the list of available DataNodes, and get instructions for block replication.
• Client ↔ DataNodes:
• The file is split into chunks by the client.
• The client uploads these chunks directly to the designated DataNodes.
• Similarly, for downloads, the client retrieves blocks from the DataNodes and reassembles them.
• NameNode ↔ DataNodes:
• The NameNode controls replication. It informs DataNodes where to store replicas of data blocks.
• DataNodes might periodically report status or heartbeat signals back to the NameNode.
Architectural Patterns & Design Principles
• Separation of Concerns:
• Different roles for each component (e.g., client for file handling, NameNode for metadata, DataNodes for storage).
• Distributed Communication:
• Utilizes gRPC for a clear, well-defined interface and message-passing between nodes.
• High Availability & Fault Tolerance:
• Block replication across multiple DataNodes ensures fault tolerance.
• Modularity:
• Each component can be developed, deployed, and even scaled independently.