# tekshteint/distributed-storage architecture

> Architecture diagram of the GitHub repository tekshteint/distributed-storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tekshteint/distributed-storage
- Repository: https://github.com/tekshteint/distributed-storage

## Overview

##### Project type and purpose
- This is a distributed storage system (a backend/distributed‐services project).
- The Master node orchestrates file uploads/downloads, splits files into chunks, and delegates storage to multiple StorageNodes.
- StorageNodes are simple Python services that persist file chunks.

##### File structure
- Top-level:
- Master/  – Dockerfile, requirements.txt, server.py (the master service)
- StorageNodes/  – Dockerfile, requirements.txt, storage.py (the storage service)
- docker-compose.yml – brings up one Master container and a set of StorageNode containers
- demos/ – sample workflows (GIFs)
- No frontend; this is purely service-to-service with REST endpoints.
- Each service is containerized, has its own Python dependencies.

##### README
- Describes endpoints:
- POST /upload → returns file UUID
- GET  /download/<fileID> → reassembles and returns the file
- Explains file splitting across storage nodes.
- Demonstrates use via Docker Compose.

##### Main components & interactions
a. Components:
- Client (e.g., curl or HTTP client)
- Master Service (Master/server.py) in a Docker container
- N StorageNode Services (StorageNodes/storage.py) in separate Docker containers
- Docker-Compose network facilitating inter-container communication
b. Interactions:
1. Client → Master: upload POST /upload (file in body)
2. Master:
- Splits file into k chunks
- For each chunk, issues POST /store to different StorageNodes
3. StorageNode: persists chunk, responds with acknowledgement
4. Master aggregates storage‐locations, returns file UUID to client
5. Client → Master: GET /download/<UUID>
6. Master fetches chunks from StorageNodes, reassembles file, returns file data
c. Architectural patterns:
- Service-oriented architecture (Master as coordinator, stateless StorageNodes)
- Distributed file‐chunk storage (no single point of data storage aside from Master metadata)
- Dockerized microservices on a shared overlay network
d. Key technologies:
- Python (Flask or similar HTTP framework)
- Docker & Docker Compose
- REST API
- File I/O & serialization in storage.py

##### Detailed element checklist
- Components: Client, Master, StorageNode1…N
- Data flows: upload, chunk dispatch, chunk ack, download request, chunk fetch, file reassembly
- Labels: service names, file paths (e.g., Master/server.py), ports (9753, etc.)
- Docker Compose boundary to indicate orchestration

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client["HTTP Client\n(CLI/Browser)"]:::client
    Client -->|"POST /upload"| MasterSvc

    subgraph "Docker Compose"
        style DockerCompose fill:none,stroke:#333,stroke-dasharray:5 5

        MasterSvc["Master Service\n(server.py)\nPort 9753"]:::master
        MasterSvc -->|"Splits into k chunks\nPOST /store"| SN1
        MasterSvc -->|"Splits into k chunks\nPOST /store"| SN2
        MasterSvc -->|"Splits into k chunks\nPOST /store"| SN3

        SN1["StorageNode 1\n(storage.py)\nPort 9754"]:::storage
        SN2["StorageNode 2\n(storage.py)\nPort 9755"]:::storage
        SN3["StorageNode N\n(storage.py)\nPort 9756"]:::storage

        SN1 -->|"ACK"| MasterSvc
        SN2 -->|"ACK"| MasterSvc
        SN3 -->|"ACK"| MasterSvc

        Client -->|"GET /download/<fileID>"| MasterSvc

        MasterSvc -->|"GET /chunk/<chunkID>"| SN1
        MasterSvc -->|"GET /chunk/<chunkID>"| SN2
        MasterSvc -->|"GET /chunk/<chunkID>"| SN3
        SN1 -->|"Chunk Data"| MasterSvc
        SN2 -->|"Chunk Data"| MasterSvc
        SN3 -->|"Chunk Data"| MasterSvc
    end

    subgraph "Project Files" 
        direction TB
        MDocker["Dockerfile"]:::compose
        MReq["requirements.txt"]:::compose
        SDocker["StorageNodes/Dockerfile"]:::compose
        SReq["requirements.txt"]:::compose
        ComposeYML["docker-compose.yml"]:::compose
        Demos["demos/"]:::compose
    end

    click MasterSvc "https://github.com/tekshteint/distributed-storage/blob/main/Master/server.py"
    click MDocker "https://github.com/tekshteint/distributed-storage/tree/main/Master/Dockerfile"
    click MReq "https://github.com/tekshteint/distributed-storage/blob/main/Master/requirements.txt"
    click SN1 "https://github.com/tekshteint/distributed-storage/blob/main/StorageNodes/storage.py"
    click SN2 "https://github.com/tekshteint/distributed-storage/blob/main/StorageNodes/storage.py"
    click SN3 "https://github.com/tekshteint/distributed-storage/blob/main/StorageNodes/storage.py"
    click SDocker "https://github.com/tekshteint/distributed-storage/tree/main/StorageNodes/Dockerfile"
    click SReq "https://github.com/tekshteint/distributed-storage/blob/main/StorageNodes/requirements.txt"
    click ComposeYML "https://github.com/tekshteint/distributed-storage/blob/main/docker-compose.yml"
    click Demos "https://github.com/tekshteint/distributed-storage/tree/main/demos/"

    classDef client fill:#cccccc,stroke:#333,stroke-width:1px
    classDef master fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef compose fill:#f2f2f2,stroke:#333,stroke-width:1px
```
