# hashicorp/serf architecture

> Architecture diagram of the GitHub repository hashicorp/serf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hashicorp/serf
- Repository: https://github.com/hashicorp/serf

## Overview

##### Project type and purpose
- This is a masterless, decentralized service‐discovery and orchestration system written in Go.
- It has two “products”:
a) The Serf library (package “serf” plus subpackages) implements a lightweight gossip protocol, membership, event system, failure detection, conflict resolution, and an RPC API.
b) The Serf CLI agent (`cmd/serf`) wraps the library to provide a command‐line interface for running agents, joining clusters, emitting and subscribing to events, queries, key generation, etc.

##### File structure
Top‐level directories and their roles:
- cmd/serf: the main CLI process, subcommands (agent, join, leave, event, query, keygen, etc.) and their tests.
- serf: the core library implementing gossip (broadcast, coalesce, delegate patterns), membership, messages, queries, snapshots, key management.
- client: a Go RPC client wrapper to talk to a Serf agent over the agent’s RPC API.
- coordinate: a higher‐level coordinator or distributed‐lock abstraction built on Serf events.
- demo: opinionated example clusters (Vagrant, load‐balancer, user data scripts) showing how to deploy and wire Serf in real infra.
- docs: markdown files describing agent internals, commands, recipes, upgrade paths.
- ops‐misc: packaging and init scripts for Debian, upstart, systemd, macOS plist.
- scripts: build and CI helper scripts (Docker builder, dist, cluster‐startup).
- testutil: small utilities (retry loops, test logs) used across packages.
- version: single‐source version information.

##### README for additional insights
- Serf agents communicate over UDP (gossip) and expose a TCP RPC API (`rpc-addr`) for control/queries.
- It supports a plugin/event‐handler model on top of the gossip stream.
- Metrics emission is pluggable via build tags (`armonmetrics` or `hashicorpmetrics`).

- Serf CLI (cmd/serf)
- RPC Client (`client/rpc_client.go`)
- Serf Agent (main library `serf` package + supporting packages)
- Gossip Transport (UDP mesh via memberlist)
- Delegate/Event System (event handlers, user events, telemetry)
- Query subsystem (RPC queries over the gossip mesh)
- Failure detector and membership manager
- Key management and encryption
- Coordinate module (higher‐level lock/election)
- Metrics subsystem (go‐metrics or armon‐metrics)
- Packaging & Deployment (ops‐misc init scripts, Dockerfile, Vagrant)
- Demo clusters (Vagrant, load-balancer)

- CLI → RPC Client → Serf Agent’s RPC Server
- Serf Agent ↔ Gossip Transport ↔ Serf Agent (other nodes in cluster)
- Gossip events → Delegate → Event Handlers → User‐supplied plugins or coordinate module
- Serf Agent → Metrics subsystem → External metrics backend
- Coordinate module subscribes to Serf Agent events and issues commands via RPC client or library API
- Demo scripts spin up multiple Serf Agents, wire them via join commands and configure load balancer via events

c. Architectural patterns and design principles:
- Peer-to-peer gossip for membership (no single point of failure)
- Pub/Sub-style event system on top of gossip
- Clean separation: core library vs. CLI vs. packaging vs. examples
- Pluggable metrics and event handlers (via Go interfaces and build tags)
- Masterless orchestration, eventual consistency

d. Key technologies/frameworks:
- Go modules and standard library (net/rpc, UDP, crypto)
- Memberlist (gossip mesh, coalesce)
- Go testing and testify for unit tests
- GNU Make, shell scripts, Docker, Vagrant for CI and demos

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI and Client Layer
    subgraph "CLI Layer"
        direction TB
        CLI["Serf CLI Agent"]:::cli
        RPCClient["RPC Client Wrapper"]:::client
    end

    %% Core Library
    subgraph "Serf Library Core"
        direction TB
        LibraryCore["Library Core"]:::core
        GossipTransport["Gossip Transport"]:::core
        DelegateSystem["Delegate/Event System"]:::core
        Membership["Membership & Failure Detector"]:::core
        QuerySubsystem["Query Subsystem"]:::core
        KeyManagement["Key Management & Encryption"]:::core
        SnapshotHandling["Snapshot Handling"]:::core
    end

    %% Extension Modules
    subgraph "Extension Modules"
        direction TB
        CoordinateModule["Coordinate Module"]:::ext
        MetricsSubsystem["Metrics Subsystem"]:::ext
    end

    %% Supporting Artifacts
    subgraph "Supporting Artifacts"
        direction TB
        Packaging["Packaging & Deployment"]:::artifact
        Demo["Demo Clusters & Examples"]:::artifact
        Scripts["Build & CI Helper Scripts"]:::artifact
        Docs["Documentation"]:::artifact
        TestUtil["Test Utilities"]:::artifact
        VersionInfo["Version Information"]:::artifact
    end

    %% Relationships
    CLI -->|"uses RPC"| RPCClient
    RPCClient -->|"RPC calls"| LibraryCore
    LibraryCore -->|"gossip mesh"| GossipTransport
    GossipTransport -->|"heartbeat & messages"| LibraryCore
    LibraryCore -->|"emit events"| DelegateSystem
    DelegateSystem -->|"handles events"| CoordinateModule
    LibraryCore -->|"process queries"| QuerySubsystem
    LibraryCore -->|"manage keys"| KeyManagement
    LibraryCore -->|"snapshot state"| SnapshotHandling
    LibraryCore -->|"send metrics"| MetricsSubsystem
    Demo -->|"deploy agents"| CLI
    Packaging -->|"init scripts"| CLI
    Scripts -->|"build & test"| CLI
    Docs -->|"describe usage"| CLI
    TestUtil -->|"test helpers"| LibraryCore
    TestUtil -->|"test helpers"| CoordinateModule
    VersionInfo -->|"version data"| LibraryCore

    %% Click Events
    click CLI "https://github.com/hashicorp/serf/tree/master/cmd/serf/"
    click RPCClient "https://github.com/hashicorp/serf/blob/master/client/rpc_client.go"
    click LibraryCore "https://github.com/hashicorp/serf/tree/master/serf/"
    click GossipTransport "https://github.com/hashicorp/serf/blob/master/serf/broadcast.go"
    click DelegateSystem "https://github.com/hashicorp/serf/blob/master/serf/delegate.go"
    click Membership "https://github.com/hashicorp/serf/blob/master/serf/serf.go"
    click QuerySubsystem "https://github.com/hashicorp/serf/blob/master/serf/query.go"
    click KeyManagement "https://github.com/hashicorp/serf/blob/master/serf/keymanager.go"
    click SnapshotHandling "https://github.com/hashicorp/serf/blob/master/serf/snapshot.go"
    click CoordinateModule "https://github.com/hashicorp/serf/blob/master/coordinate/coordinate.go"
    click MetricsSubsystem "https://github.com/hashicorp/serf/blob/master/serf/event.go"
    click Packaging "https://github.com/hashicorp/serf/tree/master/ops-misc/"
    click Demo "https://github.com/hashicorp/serf/tree/master/demo/"
    click Scripts "https://github.com/hashicorp/serf/tree/master/scripts/"
    click Docs "https://github.com/hashicorp/serf/tree/master/docs/"
    click TestUtil "https://github.com/hashicorp/serf/tree/master/testutil/"
    click VersionInfo "https://github.com/hashicorp/serf/blob/master/version/version.go"

    %% Styles
    classDef cli fill:#a2d5c6,stroke:#333,stroke-width:1px
    classDef client fill:#b7e4c7,stroke:#333,stroke-width:1px
    classDef core fill:#74c0fc,stroke:#333,stroke-width:1px
    classDef ext fill:#ffd43b,stroke:#333,stroke-width:1px
    classDef artifact fill:#adb5bd,stroke:#333,stroke-width:1px
```
