# rajender76/haystack-dynamic-replication architecture

> Architecture diagram of the GitHub repository rajender76/haystack-dynamic-replication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rajender76/haystack-dynamic-replication
- Repository: https://github.com/rajender76/haystack-dynamic-replication

## Overview

Haystack is a distributed photo-storage system. A client uploads by requesting a store allocation from the directory, writing the photo to that store, then registering its location; downloads use the cache first and fall back to directory locations and stores. The directory coordinates metadata, leader election and store health, while the replication manager audits replica counts and schedules replication and de-replication. The README documents several cross-service flows not visible in the partial excerpts; those relationships are cited to the README.

Main entry points: [`directory/directory_service.py`](https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py) (Directory Service), [`store/store_service.py`](https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py) (Store Service), [`client/client_service.py`](https://github.com/rajender76/haystack-dynamic-replication/blob/main/client/client_service.py) (Client CLI), [`replication/replication_manager.py`](https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py) (Replication Manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_access["Client Access"]
  node_client["Client CLI<br/>[client_service.py]"]
  node_nginx["Nginx Load Balancer<br/>[nginx.conf]"]
  node_cache["Cache Service<br/>[cache_service.py]"]
end

subgraph group_metadata["Metadata Coordination"]
  node_directory["Directory Service"]
  node_metadata_state[("Metadata and WAL")]
  node_leader_election["Leader Election"]
  node_store_registry["Store Health Registry"]
end

subgraph group_storage["Photo Storage"]
  node_store["Store Service<br/>[store_service.py]"]
  node_volumes[("Needle Volumes<br/>[store_service.py]")]
  node_access_tracking["Access Tracking<br/>[store_service.py]"]
end

subgraph group_replication["Replication Management"]
  node_replication_manager["Replication Manager"]
  node_replica_policy["Replica Audit and Policy"]
  node_task_queues["Replication Task Queues"]
end

node_user(("Photo User"))
node_redis[("Redis<br/>[redis.conf]")]

node_user -->|"uploads or downloads"| node_client
node_client -->|"routes directory requests"| node_nginx
node_nginx -->|"balances requests"| node_directory
node_client -->|"allocates and registers"| node_directory
node_client -->|"writes or reads"| node_store
node_client -->|"checks cached photo"| node_cache
node_cache -->|"reads and writes cache"| node_redis
node_directory -->|"updates metadata"| node_metadata_state
node_directory -->|"elects leader via"| node_leader_election
node_leader_election -->|"stores leadership"| node_redis
node_directory -->|"filters healthy locations"| node_store_registry
node_store -->|"reads and writes needles"| node_volumes
node_store -.->|"pushes photo cache data"| node_redis
node_store -->|"registers and reports health"| node_directory
node_store -->|"records access rates"| node_access_tracking
node_access_tracking -->|"reports access statistics"| node_replication_manager
node_directory -.->|"notifies photo changes"| node_replication_manager
node_replication_manager -->|"evaluates replica needs"| node_replica_policy
node_replication_manager -->|"reads locations and registers replicas"| node_directory
node_replication_manager -->|"enqueues work"| node_task_queues
node_task_queues -->|"replicates or deletes photo"| node_store
node_replica_policy -->|"audits photo listings"| node_directory

click node_client "https://github.com/rajender76/haystack-dynamic-replication/blob/main/client/client_service.py"
click node_nginx "https://github.com/rajender76/haystack-dynamic-replication/blob/main/nginx/nginx.conf"
click node_cache "https://github.com/rajender76/haystack-dynamic-replication/blob/main/cache/cache_service.py"
click node_redis "https://github.com/rajender76/haystack-dynamic-replication/blob/main/redis/redis.conf"
click node_directory "https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py"
click node_metadata_state "https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py"
click node_leader_election "https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py"
click node_store_registry "https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py"
click node_store "https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py"
click node_volumes "https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py"
click node_access_tracking "https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py"
click node_replication_manager "https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py"
click node_replica_policy "https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py"
click node_task_queues "https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client,node_nginx,node_cache,node_user toneBlue
class node_directory,node_metadata_state,node_leader_election,node_store_registry,node_redis toneAmber
class node_store,node_volumes,node_access_tracking toneMint
class node_replication_manager,node_replica_policy,node_task_queues toneRose
```

## Components

### Client Access

- [Client CLI](https://github.com/rajender76/haystack-dynamic-replication/blob/main/client/client_service.py): `client/client_service.py`
- [Nginx Load Balancer](https://github.com/rajender76/haystack-dynamic-replication/blob/main/nginx/nginx.conf): `nginx/nginx.conf`
- [Cache Service](https://github.com/rajender76/haystack-dynamic-replication/blob/main/cache/cache_service.py): `cache/cache_service.py`

### Metadata Coordination

- [Directory Service](https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py): `directory/directory_service.py`
- [Metadata and WAL](https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py): `directory/directory_service.py`
- [Leader Election](https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py): `directory/directory_service.py`
- [Store Health Registry](https://github.com/rajender76/haystack-dynamic-replication/blob/main/directory/directory_service.py): `directory/directory_service.py`

### Photo Storage

- [Store Service](https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py): `store/store_service.py`
- [Needle Volumes](https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py): `store/store_service.py`
- [Access Tracking](https://github.com/rajender76/haystack-dynamic-replication/blob/main/store/store_service.py): `store/store_service.py`

### Replication Management

- [Replication Manager](https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py): `replication/replication_manager.py`
- [Replica Audit and Policy](https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py): `replication/replication_manager.py`
- [Replication Task Queues](https://github.com/rajender76/haystack-dynamic-replication/blob/main/replication/replication_manager.py): `replication/replication_manager.py`

### Other components

- **Photo User**
- [Redis](https://github.com/rajender76/haystack-dynamic-replication/blob/main/redis/redis.conf): `redis/redis.conf`

## Connections

- Photo User → Client CLI: uploads or downloads (evidence: `README.md`)
- Client CLI → Nginx Load Balancer: routes directory requests (evidence: `README.md`)
- Nginx Load Balancer → Directory Service: balances requests (evidence: `README.md`)
- Client CLI → Directory Service: allocates and registers (evidence: `client/client_service.py`)
- Client CLI → Store Service: writes or reads (evidence: `client/client_service.py`)
- Client CLI → Cache Service: checks cached photo (evidence: `README.md`)
- Cache Service → Redis: reads and writes cache (evidence: `cache/cache_service.py`)
- Directory Service → Metadata and WAL: updates metadata (evidence: `directory/directory_service.py`)
- Directory Service → Leader Election: elects leader via (evidence: `directory/directory_service.py`)
- Leader Election → Redis: stores leadership (evidence: `directory/directory_service.py`)
- Directory Service → Store Health Registry: filters healthy locations (evidence: `directory/directory_service.py`)
- Store Service → Needle Volumes: reads and writes needles (evidence: `store/store_service.py`)
- Store Service → Redis: pushes photo cache data (evidence: `README.md`)
- Store Service → Directory Service: registers and reports health (evidence: `store/store_service.py`)
- Store Service → Access Tracking: records access rates (evidence: `store/store_service.py`)
- Access Tracking → Replication Manager: reports access statistics (evidence: `README.md`)
- Directory Service → Replication Manager: notifies photo changes (evidence: `README.md`)
- Replication Manager → Replica Audit and Policy: evaluates replica needs (evidence: `replication/replication_manager.py`)
- Replication Manager → Directory Service: reads locations and registers replicas (evidence: `replication/replication_manager.py`)
- Replication Manager → Replication Task Queues: enqueues work (evidence: `replication/replication_manager.py`)
- Replication Task Queues → Store Service: replicates or deletes photo (evidence: `replication/replication_manager.py`)
- Replica Audit and Policy → Directory Service: audits photo listings (evidence: `replication/replication_manager.py`)
