How tekshteint/distributed-storage works
Project type and purpose
• This is a distributed storage system (a backend/distributed‐services project).
• The Master node orchestrates file uploads/downloads, splits files into chunks, and delegates storage to multiple StorageNodes.
• StorageNodes are simple Python services that persist file chunks.
File structure
• Top-level:
• Master/ – Dockerfile, requirements.txt, server.py (the master service)
• StorageNodes/ – Dockerfile, requirements.txt, storage.py (the storage service)
• docker-compose.yml – brings up one Master container and a set of StorageNode containers
• demos/ – sample workflows (GIFs)
• No frontend; this is purely service-to-service with REST endpoints.
• Each service is containerized, has its own Python dependencies.
README
• Describes endpoints:
• POST /upload → returns file UUID
• GET /download/<fileID> → reassembles and returns the file
• Explains file splitting across storage nodes.
• Demonstrates use via Docker Compose.
Main components & interactions
a. Components:
• Client (e.g., curl or HTTP client)
• Master Service (Master/server.py) in a Docker container
• N StorageNode Services (StorageNodes/storage.py) in separate Docker containers
• Docker-Compose network facilitating inter-container communication
b. Interactions:
1. Client → Master: upload POST /upload (file in body)
2. Master:
• Splits file into k chunks
• For each chunk, issues POST /store to different StorageNodes
3. StorageNode: persists chunk, responds with acknowledgement
4. Master aggregates storage‐locations, returns file UUID to client
5. Client → Master: GET /download/<UUID>
6. Master fetches chunks from StorageNodes, reassembles file, returns file data
c. Architectural patterns:
• Service-oriented architecture (Master as coordinator, stateless StorageNodes)
• Distributed file‐chunk storage (no single point of data storage aside from Master metadata)
• Dockerized microservices on a shared overlay network
d. Key technologies:
• Python (Flask or similar HTTP framework)
• Docker & Docker Compose
• REST API
• File I/O & serialization in storage.py
Detailed element checklist
• Components: Client, Master, StorageNode1…N
• Data flows: upload, chunk dispatch, chunk ack, download request, chunk fetch, file reassembly
• Labels: service names, file paths (e.g., Master/server.py), ports (9753, etc.)
• Docker Compose boundary to indicate orchestration