How rajender76/haystack-dynamic-replication works
Haystack is a distributed photo-storage system. A client uploads by requesting a store allocation from the directory, writing the photo to that store, then registering its location; downloads use the cache first and fall back to directory locations and stores. The directory coordinates metadata, leader election and store health, while the replication manager audits replica counts and schedules replication and de-replication. The README documents several cross-service flows not visible in the partial excerpts; those relationships are cited to the README.
Main entry points: directory_service.py Directory Service · store_service.py Store Service · client_service.py Client CLI · replication_manager.py Replication Manager
Components 15
Client Access
- Client CLI
client/client_service.py - Nginx Load Balancer
nginx/nginx.conf - Cache Service
cache/cache_service.py
Metadata Coordination
- Directory Service
directory/directory_service.py - Metadata and WAL
directory/directory_service.py - Leader Election
directory/directory_service.py - Store Health Registry
directory/directory_service.py
Photo Storage
- Store Service
store/store_service.py - Needle Volumes
store/store_service.py - Access Tracking
store/store_service.py
Replication Management
- Replication Manager
replication/replication_manager.py - Replica Audit and Policy
replication/replication_manager.py - Replication Task Queues
replication/replication_manager.py
Other components
- Photo User
- Redis
redis/redis.conf
Connections 22
- Photo User → Client CLI: uploads or downloads
README.md - Client CLI → Nginx Load Balancer: routes directory requests
README.md - Nginx Load Balancer → Directory Service: balances requests
README.md - Client CLI → Directory Service: allocates and registers
client/client_service.py - Client CLI → Store Service: writes or reads
client/client_service.py - Client CLI → Cache Service: checks cached photo
README.md - Cache Service → Redis: reads and writes cache
cache/cache_service.py - Directory Service → Metadata and WAL: updates metadata
directory/directory_service.py - Directory Service → Leader Election: elects leader via
directory/directory_service.py - Leader Election → Redis: stores leadership
directory/directory_service.py - Directory Service → Store Health Registry: filters healthy locations
directory/directory_service.py - Store Service → Needle Volumes: reads and writes needles
store/store_service.py - Store Service → Redis: pushes photo cache data
README.md - Store Service → Directory Service: registers and reports health
store/store_service.py - Store Service → Access Tracking: records access rates
store/store_service.py - Access Tracking → Replication Manager: reports access statistics
README.md - Directory Service → Replication Manager: notifies photo changes
README.md - Replication Manager → Replica Audit and Policy: evaluates replica needs
replication/replication_manager.py - Replication Manager → Directory Service: reads locations and registers replicas
replication/replication_manager.py - Replication Manager → Replication Task Queues: enqueues work
replication/replication_manager.py - Replication Task Queues → Store Service: replicates or deletes photo
replication/replication_manager.py - Replica Audit and Policy → Directory Service: audits photo listings
replication/replication_manager.py