# neerajjain92/distributed-shard-manager architecture

> Architecture diagram of the GitHub repository neerajjain92/distributed-shard-manager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neerajjain92/distributed-shard-manager
- Repository: https://github.com/neerajjain92/distributed-shard-manager

## Overview

##### Main Components
- Shard Manager Server: Represented by the main class (ShardManagerServer.java). This is the entry point of the server side.
- Core Sharding Logic: Include components like ShardManager, FailoverManager, and ShardPlacementPolicy as fundamental building blocks for shard allocation, failover handling, and balanced distribution.
- Data Model: Capture the model components (Shard, Host, ShardAssignment, ShardRange, etc.) to show the underlying data structures that represent shards and hosts.
- Metadata Storage: Highlight the abstraction (MetadataStore) and its ZooKeeper-based implementation (ZookeeperMetadataStore) which is used for distributed coordination and persistent metadata.
- Client Components: Include the ShardManagerClient and RoutingTableWatcher. The client is responsible for fetching and reacting to changes in the routing table.
- Example Usage: Optionally reference the example usage provided by ShardedDataService to illustrate how applications use the system.

##### Relationships and Interactions
- Directional data flow between the client and server; for instance, how the ShardManagerClient queries the server for key-based routing, and how RoutingTableWatcher listens to changes.
- ShardManager interacts closely with the FailoverManager to manage host failures.
- ShardPlacementPolicy works with ShardManager to decide on optimal shard distribution.

##### Architectural Patterns and Design Principles
- Distributed Coordination: Emphasize ZooKeeper’s role in ensuring consistency among nodes.
- Modular Design: The separation of concerns between client and server as well as distinct packages for core logic, client operations, data modeling, and metadata storage.
- Fault Tolerance and Auto-Recovery: The FailoverManager automatically manages host failures and promotes a backup (SECONDARY) host to PRIMARY.
- Dynamic Load Balancing: Indicate that the system continuously rebalances shards based on host load and capacity.

##### Relevant Technologies, Frameworks, and Libraries
- Java: Since the entire project is built in Java.
- ZooKeeper: Used for distributed coordination and metadata storage (explicitly seen in ZookeeperMetadataStore).
- Maven: Denoted by the pom.xml file for build and dependency management.

a. Start by drawing the central ShardManager module along with FailoverManager and ShardPlacementPolicy adjacent to it. Group these to represent the core business logic.
d. Data Model as supplementary blocks to illustrate how shards and host information is structured, linking them to the ShardManager components.
e. Interactions. For example, show how failover is handled by the FailoverManager when detected by a host monitoring process (even if implicitly via ZooKeeper notifications).

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Server Side"
        SManagerServer["Shard Manager Server"]:::server
        subgraph "Core Components"
            ShardManager["ShardManager"]:::core
            FailoverManager["FailoverManager"]:::core
            ShardPlacementPolicy["ShardPlacementPolicy"]:::core
        end
        subgraph "Metadata Storage"
            MetadataStore["MetadataStore"]:::storage
            ZookeeperMetadataStore["ZookeeperMetadataStore"]:::storage
        end
    end

    subgraph "Data Model"
        Shard["Shard"]:::model
        Host["Host"]:::model
        ShardAssignment["ShardAssignment"]:::model
        ShardRange["ShardRange"]:::model
    end

    subgraph "External Service"
        ZooKeeper["ZooKeeper"]:::external
    end

    subgraph "Client Side"
        ShardManagerClient["ShardManagerClient"]:::client
        RoutingTableWatcher["RoutingTableWatcher"]:::client
        ShardedDataService["ShardedDataService"]:::client
    end

    SManagerServer -->|"initiates"| ShardManager
    ShardManager -->|"handles_failover"| FailoverManager
    ShardManager -->|"applies_policy"| ShardPlacementPolicy
    ShardManager -->|"manages"| Shard
    ShardManager -->|"manages"| Host
    ShardManager -->|"manages"| ShardAssignment
    ShardManager -->|"manages"| ShardRange
    ShardManager -->|"updates_metadata"| ZookeeperMetadataStore
    ZookeeperMetadataStore -->|"implements"| MetadataStore
    ZookeeperMetadataStore -->|"syncs_with"| ZooKeeper
    ShardManagerClient -->|"queries"| SManagerServer
    ZooKeeper -->|"notifies"| RoutingTableWatcher
    ShardedDataService -->|"utilizes"| ShardManagerClient

    classDef server fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef core fill:#a2d5f2,stroke:#333,stroke-width:2px;
    classDef storage fill:#f7a8a8,stroke:#333,stroke-width:2px;
    classDef model fill:#b1e5d3,stroke:#333,stroke-width:2px;
    classDef external fill:#f9d56e,stroke:#333,stroke-width:2px;
    classDef client fill:#c3e88d,stroke:#333,stroke-width:2px;

    click SManagerServer "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/ShardManagerServer.java"
    click ShardManager "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/core/ShardManager.java"
    click FailoverManager "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/core/FailoverManager.java"
    click ShardPlacementPolicy "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/core/ShardPlacementPolicy.java"
    click Shard "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/model/Shard.java"
    click Host "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/model/Host.java"
    click ShardAssignment "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/model/ShardAssignment.java"
    click ShardRange "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/model/ShardRange.java"
    click MetadataStore "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/store/MetadataStore.java"
    click ZookeeperMetadataStore "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/store/ZookeeperMetadataStore.java"
    click ShardManagerClient "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/client/ShardManagerClient.java"
    click RoutingTableWatcher "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/client/RoutingTableWatcher.java"
    click ShardedDataService "https://github.com/neerajjain92/distributed-shard-manager/blob/main/src/main/java/com/meta/shardmanager/example/ShardedDataService.java"
```
