# bmuller/kademlia architecture

> Architecture diagram of the GitHub repository bmuller/kademlia, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bmuller/kademlia
- Repository: https://github.com/bmuller/kademlia

## Overview

##### Project Type and Purpose
- This is a Python library implementing the Kademlia Distributed Hash Table (DHT) protocol.
- It provides an asynchronous, peer-to-peer key/value store based on the Kademlia paper, using asyncio and RPC-over-UDP for communication.
- Consumers write Python scripts that instantiate a Server (node), bootstrap into the DHT, and call get()/set() on the network.

##### File Structure Analysis
- Top-level directories:
- kademlia/             ← core implementation modules
- examples/             ← sample scripts showing node startup, bootstrap, get/set
- docs/                 ← Sphinx documentation
- .github/, CI configs, pytest.ini, pyproject.toml → CI/build/test config
- Inside kademlia/:
- network.py            ← Server class, asyncio UDP transport, bootstrap logic
- node.py               ← RPC message handlers for GET/STORE/LOOKUP
- routing.py            ← K-bucket routing table, node lookups
- storage.py            ← Local key/value store abstraction
- crawling.py           ← Iterative node lookup (“FIND_NODE”, “FIND_VALUE”)
- protocol.py           ← Message formats, RPC call/response logic
- utils.py              ← ID generation, distance metric
- Tests mirror these modules to validate node behavior, routing, storage, serialization, RPC.

##### README Insights
- Emphasizes alignment with the original Kademlia paper.
- Points out use of asyncio and RPC/UDP to enable NAT traversal.
- Shows simple user API: Server.listen, Server.bootstrap, Server.set, Server.get.

a. Main Components:
- User Application (example scripts)
- Asyncio Event Loop
- network.Server (UDP transport, bootstrap logic)
- protocol.RPCProtocol (serialization of RPC requests/responses)
- node.Node (RPC request handlers)
- routing.RoutingTable (K-buckets, peer management)
- crawling.Crawler (iterative FIND_NODE/FIND_VALUE logic)
- storage.Storage (local key/value store)
- External Peers (other Kademlia nodes over Internet/UDP)
b. Relationships & Data Flow:
1. User code calls Server.listen() → binds UDP port in asyncio loop.
2. User calls Server.bootstrap() → RPCProtocol sends PING to bootstrap peers → routing table updated.
3. set(key,value): Server invokes storage.Store locally and RPC STORE to k closest nodes via Crawler.
4. get(key): Server/Crawler issues iterative FIND_VALUE/FIND_NODE messages → RPCProtocol → network I/O → routing updates → Returns value.
5. Incoming RPCs are demultiplexed in protocol.RPCProtocol → node.Node handlers → routing & storage modules.
c. Architectural Patterns & Design Principles:
- Modular layering: Network layer (Server & RPCProtocol), Overlay logic (Node, Crawler, Routing), Storage layer.
- Asynchronous, event-driven design using asyncio’s event loop.
- Single-process, single-threaded concurrency via non-blocking UDP.
- Separation of concerns: routing vs storage vs transport vs message formatting.
d. Key Technologies & Frameworks:
- Python 3 asyncio (event loop, transports)
- UDP via asyncio DatagramProtocol
- rpcudp library for RPC-over-UDP semantics
- Sphinx for docs, pytest for testing

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Layer"
        User["User Application\n(examples/)"]:::user
    end

    subgraph "Async Layer"
        Async["Asyncio Event Loop"]:::infra
    end

    subgraph "Network Layer"
        Server["Server"]:::network
        RPC["RPCProtocol"]:::network
        UDP["UDP Transport"]:::network
    end

    subgraph "Overlay Logic"
        Node["Node Handlers"]:::overlay
        Routing["RoutingTable"]:::overlay
        Crawler["Crawler"]:::overlay
    end

    subgraph "Storage Layer"
        Storage["Storage"]:::storage
    end

    subgraph "Utilities"
        Utils["Utils"]:::utils
    end

    subgraph "External"
        Peers["External Peers\n(Internet/UDP)"]:::infra
    end

    subgraph "Support"
        Tests["Test Suite"]:::infra
        Docs["Documentation"]:::infra
    end

    User -->|"listen(), bootstrap(), set(), get()"| Async
    Async --> Server
    Server -->|"bind UDP"| UDP
    Server -->|"calls RPC"| RPC
    RPC -->|"serialize RPC"| UDP
    UDP -->|"sends PING,STORE,FIND_NODE,FIND_VALUE"| Peers
    Peers --> UDP
    UDP --> RPC
    RPC --> Node
    Node --> Routing
    Node --> Storage
    Node --> Crawler
    Node --> Utils
    Routing --> RPC
    Routing --> Utils
    Crawler --> RPC
    Crawler --> Utils

    click User "https://github.com/bmuller/kademlia/tree/master/examples/"
    click Server "https://github.com/bmuller/kademlia/blob/master/kademlia/network.py"
    click RPC "https://github.com/bmuller/kademlia/blob/master/kademlia/protocol.py"
    click Node "https://github.com/bmuller/kademlia/blob/master/kademlia/node.py"
    click Routing "https://github.com/bmuller/kademlia/blob/master/kademlia/routing.py"
    click Crawler "https://github.com/bmuller/kademlia/blob/master/kademlia/crawling.py"
    click Storage "https://github.com/bmuller/kademlia/blob/master/kademlia/storage.py"
    click Utils "https://github.com/bmuller/kademlia/blob/master/kademlia/utils.py"
    click Tests "https://github.com/bmuller/kademlia/tree/master/kademlia/tests/"
    click Docs "https://github.com/bmuller/kademlia/blob/master/docs/source/kademlia.rst"

    classDef user fill:#a2fca0,stroke:#333,stroke-width:1px
    classDef overlay fill:#a0c4ff,stroke:#333,stroke-width:1px
    classDef network fill:#ffad5a,stroke:#333,stroke-width:1px
    classDef storage fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef utils fill:#d0afff,stroke:#333,stroke-width:1px
    classDef infra fill:#e0e0e0,stroke:#333,stroke-width:1px
```
