How bmuller/kademlia works
Project Type and Purpose
• This is a Python library implementing the Kademlia Distributed Hash Table (DHT) protocol.
• It provides an asynchronous, peer-to-peer key/value store based on the Kademlia paper, using asyncio and RPC-over-UDP for communication.
• Consumers write Python scripts that instantiate a Server (node), bootstrap into the DHT, and call get()/set() on the network.
File Structure Analysis
• Top-level directories:
• kademlia/ ← core implementation modules
• examples/ ← sample scripts showing node startup, bootstrap, get/set
• docs/ ← Sphinx documentation
• .github/, CI configs, pytest.ini, pyproject.toml → CI/build/test config
• Inside kademlia/:
• network.py ← Server class, asyncio UDP transport, bootstrap logic
• node.py ← RPC message handlers for GET/STORE/LOOKUP
• routing.py ← K-bucket routing table, node lookups
• storage.py ← Local key/value store abstraction
• crawling.py ← Iterative node lookup (“FIND_NODE”, “FIND_VALUE”)
• protocol.py ← Message formats, RPC call/response logic
• utils.py ← ID generation, distance metric
• Tests mirror these modules to validate node behavior, routing, storage, serialization, RPC.
README Insights
• Emphasizes alignment with the original Kademlia paper.
• Points out use of asyncio and RPC/UDP to enable NAT traversal.
• Shows simple user API: Server.listen, Server.bootstrap, Server.set, Server.get.
a. Main Components:
• User Application (example scripts)
• Asyncio Event Loop
• network.Server (UDP transport, bootstrap logic)
• protocol.RPCProtocol (serialization of RPC requests/responses)
• node.Node (RPC request handlers)
• routing.RoutingTable (K-buckets, peer management)
• crawling.Crawler (iterative FIND_NODE/FIND_VALUE logic)
• storage.Storage (local key/value store)
• External Peers (other Kademlia nodes over Internet/UDP)
b. Relationships & Data Flow:
1. User code calls Server.listen() → binds UDP port in asyncio loop.
2. User calls Server.bootstrap() → RPCProtocol sends PING to bootstrap peers → routing table updated.
3. set(key,value): Server invokes storage.Store locally and RPC STORE to k closest nodes via Crawler.
4. get(key): Server/Crawler issues iterative FIND_VALUE/FIND_NODE messages → RPCProtocol → network I/O → routing updates → Returns value.
5. Incoming RPCs are demultiplexed in protocol.RPCProtocol → node.Node handlers → routing & storage modules.
c. Architectural Patterns & Design Principles:
• Modular layering: Network layer (Server & RPCProtocol), Overlay logic (Node, Crawler, Routing), Storage layer.
• Asynchronous, event-driven design using asyncio’s event loop.
• Single-process, single-threaded concurrency via non-blocking UDP.
• Separation of concerns: routing vs storage vs transport vs message formatting.
d. Key Technologies & Frameworks:
• Python 3 asyncio (event loop, transports)
• UDP via asyncio DatagramProtocol
• rpcudp library for RPC-over-UDP semantics
• Sphinx for docs, pytest for testing