# libp2p/py-libp2p architecture

> Architecture diagram of the GitHub repository libp2p/py-libp2p, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/libp2p/py-libp2p
- Repository: https://github.com/libp2p/py-libp2p

## Overview

##### Project Type and Purpose
- This is an open-source Python library (“py-libp2p”) implementing the libp2p peer-to-peer networking stack.
- It provides pluggable transports (TCP, QUIC, WebSocket…), secure channels (Noise, Secio, plaintext), stream multiplexers (yamux, mplex), discovery (bootstrap, mDNS, random‐walk), peer routing (Kademlia DHT), pub/sub (floodsub, gossipsub), NAT traversal (circuit relay v2, autonat), and core utilities (ping, identify, peerstore, crypto).

##### File Structure Highlights
- libp2p/… – all core modules organized by function: transport, security, stream_muxer, discovery, pubsub, relay, kad_dht, host, identity, crypto, peer, utils, tools.
- examples/… – reference scripts demonstrating protocols.
- docs/… – ReStructuredText docs that include architectural descriptions and code examples.
- tests/… – unit and integration tests against each module.
- scripts/, Makefile, pyproject.toml – build, CI/CD, packaging.

##### Core Architectural Components
a. Host
- High-level orchestrator presenting the user API.
- Coordinates Transport, Upgrader, Discovery, Routing, Pub/Sub, Relay, Peerstore.
b. Transport Layer
- TCP (✅), QUIC/WebSocket prototypes.
- Exposes a raw “Connection” abstraction.
c. Upgrader
- Takes a raw Connection, negotiates Security and Muxing, returns a secure-multiplexed channel.
d. Security
- Insecure/plaintext, Noise, Secio; handles encryption, key exchange, session setup.
e. Stream Multiplexer
- Yamux, Mplex; multiplexes multiple logical streams over one secure connection.
f. Streams & Protocol Handlers
- Streams are opened over the upgraded channel; each stream selects a protocol ID.
- Built-in handlers: ping, identify, pubsub (floodsub/gossipsub), kad_dht, autonat, relay/circuit_v2, mdns, bootstrap, random-walk.
g. Peerstore & Crypto
- Identities, key management, peer records serialization via protobuf.
h. Tools & Services
- Async service orchestration (trio), timed caches, logging, varint, utilities.

##### Relationships & Data Flow
- Transport → raw Connection → Upgrader
- Upgrader applies Security → secure Connection → Muxer
- Muxer opens Streams → routed to Protocol Handlers
- Protocol Handlers interact with Peerstore, DHT, Discovery, Pub/Sub, Relay, Autonat
- Discovery modules seed Peerstore; DHT populates routing table; Pub/Sub and Relay build on top of streams.
- All modules communicate via async service loops (Trio).

##### Architectural Patterns & Principles
- Modular + pluggable: new transports, security, muxers, protocols can be dropped in.
- Layered abstraction: Transport → Upgrader → Muxer → Streams → Protocols.
- Separation of concerns: discovery vs. routing vs. secure channel vs. multiplexing.
- Asynchronous event loop (Trio) and service abstraction.

##### Essential Elements to Capture
- Host orchestration layer.
- Transport → Upgrader → Security → Muxer pipeline.
- Stream opening and protocol negotiation.
- Major protocols: identify, ping, pubsub, kad_dht, relay, autonat, discovery.
- Peerstore and crypto module interactions.
- Async service loop (Trio) underpinning all subsystems.
- Extension/plugin boundary around each module group.

## Diagram (Mermaid)

```mermaid
flowchart TD
    direction TB

    %% Host Layer
    subgraph "Host Layer"
        direction TB
        BasicHost["Basic Host API"]:::runtime
        RoutedHost["Routed Host (DHT)"]:::runtime
        HostDefaults["Host Defaults"]:::runtime
        HostExceptions["Host Exceptions"]:::runtime
        AutoNATService["AutoNAT Service"]:::runtime
    end

    %% Transport & Network Layer
    subgraph "Transport & Network" 
        direction TB
        TCP["TCP Transport"]:::blue
        WS["WebSocket Prototype"]:::blue
        QUIC["QUIC Prototype"]:::blue
    end

    %% Upgrader
    Upgrader["Upgrader"]:::runtime

    %% Security Plugins
    subgraph "Security Plugins"
        direction TB
        Insecure["Plaintext Transport"]:::green
        Noise["Noise Protocol"]:::green
        Secio["Secio Protocol"]:::green
        SecBaseSession["base_session"]:::green
        SecBaseTransport["base_transport"]:::green
        SecureSession["secure_session"]:::green
        SecMultistream["security_multistream"]:::green
    end

    %% Muxer Plugins
    subgraph "Muxer Plugins"
        direction TB
        Mplex["Mplex Muxer"]:::green
        MplexStream["Mplex Stream"]:::green
        Yamux["Yamux Muxer"]:::green
        MuxerMultistream["multistream negotiator"]:::green
    end

    %% Streams & Protocol Handlers
    subgraph "Streams & Protocol Handlers"
        direction TB
        Ping["Ping Protocol"]:::orange
        Identify["Identify Protocol"]:::orange
        PubSubCore["PubSub Core"]:::orange
        FloodSub["FloodSub"]:::orange
        GossipSub["GossipSub"]:::orange
        DHTCore["Kademlia DHT"]:::orange
        Relay["Circuit Relay v2"]:::orange
        DCUTR["DCUTR NAT Traversal"]:::orange
        AutoNATProto["AutoNAT Proto"]:::orange
        Bootstrap["Bootstrap Discovery"]:::orange
        mDNS["mDNS Discovery"]:::orange
        RandomWalk["Random-Walk Discovery"]:::orange
    end

    %% Peerstore & Crypto
    subgraph "Peerstore & Crypto"
        direction TB
        Peerstore["Peerstore"]:::gray
        PeerRecord["Peer Record"]:::gray
        ID["Peer ID"]:::gray
        Envelope["Peer Envelope"]:::gray
        ED25519["ED25519"]:::gray
        RSA["RSA"]:::gray
        X25519["X25519"]:::gray
    end

    %% Async Service
    Trio["Async Service Loop (Trio)"]:::gray

    %% Host connections
    BasicHost -->|"orchestrates"| TCP
    BasicHost -->|"orchestrates"| Upgrader
    BasicHost -->|"orchestrates"| Insecure
    BasicHost -->|"orchestrates"| Mplex
    BasicHost -->|"orchestrates"| Ping
    BasicHost -->|"orchestrates"| Bootstrap
    BasicHost -->|"orchestrates"| DHTCore
    BasicHost -->|"orchestrates"| Relay
    BasicHost -->|"orchestrates"| Peerstore

    %% Transport to Upgrader
    TCP -->|"raw connection"| Upgrader
    WS -->|"raw connection"| Upgrader
    QUIC -->|"raw connection"| Upgrader

    %% Upgrader pipeline
    Upgrader -->|"handshake"| Insecure
    Upgrader -->|"handshake"| Noise
    Upgrader -->|"handshake"| Secio
    Upgrader -->|"select mux"| MuxerMultistream

    %% Security to Muxer
    Insecure -->|"secure conn"| MuxerMultistream
    Noise -->|"secure conn"| MuxerMultistream
    Secio -->|"secure conn"| MuxerMultistream

    %% Muxer multiplexing to streams
    MuxerMultistream -->|"open streams"| Mplex
    MuxerMultistream -->|"open streams"| Yamux
    MuxerMultistream -->|"open streams"| MplexStream

    %% Streams to Protocol Handlers
    Mplex -->|"stream"| Ping
    Yamux -->|"stream"| Identify
    MplexStream -->|"stream"| PubSubCore
    PubSubCore -->|"uses"| FloodSub
    PubSubCore -->|"uses"| GossipSub
    PubSubCore -->|"uses"| DHTCore
    PubSubCore -->|"uses"| Relay
    PubSubCore -->|"uses"| AutoNATProto
    PubSubCore -->|"uses"| Bootstrap
    PubSubCore -->|"uses"| mDNS
    PubSubCore -->|"uses"| RandomWalk

    %% Protocol Handlers to Peerstore & Crypto
    Ping -->|"logs"| Peerstore
    Identify -->|"stores identity"| Peerstore
    DHTCore -->|"routes"| Peerstore
    Relay -->|"auth"| PeerRecord
    AutoNATProto -->|"check"| Peerstore

    %% Peerstore & Crypto usage
    Peerstore --> ID
    Peerstore --> PeerRecord
    PeerRecord --> Envelope
    ID --> ED25519
    ID --> RSA
    ID --> X25519

    %% Async service underpinning
    Trio -.-> BasicHost
    Trio -.-> Upgrader
    Trio -.-> Insecure
    Trio -.-> MuxerMultistream
    Trio -.-> Ping
    Trio -.-> DHTCore
    Trio -.-> Relay

    %% Legend
    subgraph "Legend"
        direction LR
        A["Runtime Components"]:::runtime
        B["Transport & Network"]:::blue
        C["Security & Muxing"]:::green
        D["Streams & Protocols"]:::orange
        E["Utilities & Peerstore"]:::gray
    end

    %% Click Events
    click BasicHost "https://github.com/libp2p/py-libp2p/blob/main/libp2p/host/basic_host.py"
    click RoutedHost "https://github.com/libp2p/py-libp2p/blob/main/libp2p/host/routed_host.py"
    click HostDefaults "https://github.com/libp2p/py-libp2p/blob/main/libp2p/host/defaults.py"
    click HostExceptions "https://github.com/libp2p/py-libp2p/blob/main/libp2p/host/exceptions.py"
    click AutoNATService "https://github.com/libp2p/py-libp2p/blob/main/libp2p/host/autonat/autonat.py"
    click TCP "https://github.com/libp2p/py-libp2p/blob/main/libp2p/transport/tcp/tcp.py"
    click Insecure "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/insecure/transport.py"
    click Noise "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/noise/transport.py"
    click Secio "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/secio/transport.py"
    click SecBaseSession "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/base_session.py"
    click SecBaseTransport "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/base_transport.py"
    click SecureSession "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/secure_session.py"
    click SecMultistream "https://github.com/libp2p/py-libp2p/blob/main/libp2p/security/security_multistream.py"
    click Mplex "https://github.com/libp2p/py-libp2p/blob/main/libp2p/stream_muxer/mplex/mplex.py"
    click MplexStream "https://github.com/libp2p/py-libp2p/blob/main/libp2p/stream_muxer/mplex/mplex_stream.py"
    click Yamux "https://github.com/libp2p/py-libp2p/blob/main/libp2p/stream_muxer/yamux/yamux.py"
    click MuxerMultistream "https://github.com/libp2p/py-libp2p/blob/main/libp2p/stream_muxer/muxer_multistream.py"
    click Bootstrap "https://github.com/libp2p/py-libp2p/blob/main/libp2p/discovery/bootstrap/bootstrap.py"
    click mDNS "https://github.com/libp2p/py-libp2p/blob/main/libp2p/discovery/mdns/mdns.py"
    click RandomWalk "https://github.com/libp2p/py-libp2p/blob/main/libp2p/discovery/random_walk/random_walk.py"
    click DHTCore "https://github.com/libp2p/py-libp2p/blob/main/libp2p/kad_dht/kad_dht.py"
    click Peerstore "https://github.com/libp2p/py-libp2p/blob/main/libp2p/peer/peerstore.py"
    click PeerRecord "https://github.com/libp2p/py-libp2p/blob/main/libp2p/peer/peer_record.py"
    click ID "https://github.com/libp2p/py-libp2p/blob/main/libp2p/peer/id.py"
    click Envelope "https://github.com/libp2p/py-libp2p/blob/main/libp2p/peer/envelope.py"
    click ED25519 "https://github.com/libp2p/py-libp2p/blob/main/libp2p/crypto/ed25519.py"
    click RSA "https://github.com/libp2p/py-libp2p/blob/main/libp2p/crypto/rsa.py"
    click X25519 "https://github.com/libp2p/py-libp2p/blob/main/libp2p/crypto/x25519.py"
    click Upgrader "https://github.com/libp2p/py-libp2p/blob/main/libp2p/transport/upgrader.py"
    click PubSubCore "https://github.com/libp2p/py-libp2p/blob/main/libp2p/pubsub/pubsub.py"
    click FloodSub "https://github.com/libp2p/py-libp2p/blob/main/libp2p/pubsub/floodsub.py"
    click GossipSub "https://github.com/libp2p/py-libp2p/blob/main/libp2p/pubsub/gossipsub.py"
    click Trio "https://github.com/libp2p/py-libp2p/blob/main/libp2p/tools/async_service/trio_service.py"

    %% Styles
    classDef runtime fill:#f9f,stroke:#333,stroke-width:1px,rx:10,ry:10
    classDef blue fill:#ccf,stroke:#333,stroke-width:1px
    classDef green fill:#cfc,stroke:#333,stroke-width:1px
    classDef orange fill:#fc9,stroke:#333,stroke-width:1px
    classDef gray fill:#ddd,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
