How neerajjain92/distributed-shard-manager works
Main Components
• Shard Manager Server: Represented by the main class (ShardManagerServer.java). This is the entry point of the server side.
• Core Sharding Logic: Include components like ShardManager, FailoverManager, and ShardPlacementPolicy as fundamental building blocks for shard allocation, failover handling, and balanced distribution.
• Data Model: Capture the model components (Shard, Host, ShardAssignment, ShardRange, etc.) to show the underlying data structures that represent shards and hosts.
• Metadata Storage: Highlight the abstraction (MetadataStore) and its ZooKeeper-based implementation (ZookeeperMetadataStore) which is used for distributed coordination and persistent metadata.
• Client Components: Include the ShardManagerClient and RoutingTableWatcher. The client is responsible for fetching and reacting to changes in the routing table.
• Example Usage: Optionally reference the example usage provided by ShardedDataService to illustrate how applications use the system.
Relationships and Interactions
• Directional data flow between the client and server; for instance, how the ShardManagerClient queries the server for key-based routing, and how RoutingTableWatcher listens to changes.
• ShardManager interacts closely with the FailoverManager to manage host failures.
• ShardPlacementPolicy works with ShardManager to decide on optimal shard distribution.
Architectural Patterns and Design Principles
• Distributed Coordination: Emphasize ZooKeeper’s role in ensuring consistency among nodes.
• Modular Design: The separation of concerns between client and server as well as distinct packages for core logic, client operations, data modeling, and metadata storage.
• Fault Tolerance and Auto-Recovery: The FailoverManager automatically manages host failures and promotes a backup (SECONDARY) host to PRIMARY.
• Dynamic Load Balancing: Indicate that the system continuously rebalances shards based on host load and capacity.
Relevant Technologies, Frameworks, and Libraries
• Java: Since the entire project is built in Java.
• ZooKeeper: Used for distributed coordination and metadata storage (explicitly seen in ZookeeperMetadataStore).
• Maven: Denoted by the pom.xml file for build and dependency management.
a. Start by drawing the central ShardManager module along with FailoverManager and ShardPlacementPolicy adjacent to it. Group these to represent the core business logic.
d. Data Model as supplementary blocks to illustrate how shards and host information is structured, linking them to the ShardManager components.
e. Interactions. For example, show how failover is handled by the FailoverManager when detected by a host monitoring process (even if implicitly via ZooKeeper notifications).