# dcos/lashup architecture

> Architecture diagram of the GitHub repository dcos/lashup, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dcos/lashup
- Repository: https://github.com/dcos/lashup

## Overview

##### Main Components

- Overlay Builder:
- “lashup_hyparview” components (e.g., lashup_hyparview_membership, lashup_hyparview_ping_handler, lashup_hyparview_events). These modules manage the overlay network which forms a strongly connected, sparse TCP graph ensuring scalability and stability.
- Subcomponents like randomized timers, throttling, and persistent member storage.

- Routing & Failure Detection:
- Components such as lashup_gm, lashup_gm_events, lashup_gm_route, lashup_gm_route_events, lashup_gm_fanout, and the multicast-related modules (e.g., lashup_gm_mc, lashup_gm_mc_events) that handle dissemination of routing information and failure detection.
- Dependency where health checks, gossip messages, and adjacency table updates work together to quickly detect node failures and propagate changes.

- Distributed Key-Value Store:
- “lashup_kv*” modules (like lashup_kv, lashup_kv_aae_mgr, lashup_kv_sync_rx_fsm, lashup_kv_sync_tx_fsm) that together implement a CRDT store over riak_dt with Mnesia as the persistence layer.
- API for performing operations and the subscription functionality (pattern-based subscriptions using ETS match specs).

- Supervision & Infrastructure:
- Supervisory tree (modules such as lashup_core_sup, lashup_platform_sup, lashup_gm_sup, lashup_kv_sup, lashup_gm_worker_sup, and lashup_sup) that organizes how these components start, crash-recover, and interact.
- Utility modules (lashup_config, lashup_utils) that support configuration and shared functionality.

- From the overlay builder to the routing layer: show how the overlay’s adjacencies and membership details are communicated upward.

- Related modules into logical clusters:
- One cluster for network construction and maintenance (Overlay Builder).
- One cluster for routing and failure detection.
- One cluster for the distributed database functionality.
- An outer layer that shows the supervision infrastructure coordinating these inner components.

##### Architectural Patterns and Technologies

- Architectural patterns:
- OTP supervision hierarchies, which enforce fault tolerance and modular design.
- Use of the HyParView protocol (with modifications) for building the overlay network.
- Gossip-based dissemination for rapid failure detection and state propagation.
- CRDTs (via basho/riak_dt) for a distributed key-value store and eventual consistency.

- Technologies and frameworks:
- Erlang/OTP (including Mnesia for persistence).
- riak_dt for CRDT operations.
- ETS for in-memory subscriptions.
- CircleCI for continuous integration (as hinted by the .circleci config).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Supervision & Infrastructure
    subgraph "Supervision & Infrastructure"
        sup1["lashup_core_sup"]
        sup2["lashup_platform_sup"]
        sup3["lashup_sup"]
        sup4["lashup_gm_mc_sup"]
        sup5["lashup_config"]
        sup6["lashup_utils"]
    end

    %% Overlay Builder (HyParView-based)
    subgraph "Overlay Builder (HyParView-based)"
        hb1["lashup_hyparview_membership"]
        hb2["lashup_hyparview_ping_handler"]
        hb3["lashup_hyparview_events"]
    end

    %% Routing & Failure Detection
    subgraph "Routing & Failure Detection"
        gm1["lashup_gm"]
        gm2["lashup_gm_events"]
        gm3["lashup_gm_route"]
        gm4["lashup_gm_route_events"]
        gm5["lashup_gm_fanout"]
        gm6["lashup_gm_mc"]
        gm7["lashup_gm_mc_events"]
        gm8["lashup_gm_probe"]
        gm9["lashup_gm_sup"]
        gm10["lashup_gm_worker_sup"]
        gm11["lashup_gm_sync_worker"]
    end

    %% Distributed CRDT Store (KV)
    subgraph "Distributed CRDT Store (KV)"
        kv1["lashup_kv"]
        kv2["lashup_kv_aae_mgr"]
        kv3["lashup_kv_sync_rx_fsm"]
        kv4["lashup_kv_sync_tx_fsm"]
        kv5["lashup_kv_sup"]
        kv6["lashup_kv_aae_sup"]
    end

    %% Intergroup Relationships
    sup3 --> hb1
    sup3 --> gm1
    sup3 --> kv1

    hb1 -->|"membership_details"| gm1
    hb3 -->|"adjacency_update"| gm8

    gm2 -->|"gossip_health"| kv1
    gm6 -->|"multicast_trigger"| kv1

    %% Styles
    classDef overlay fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef routing fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef kv fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef supervision fill:#D3D3D3,stroke:#000,stroke-dasharray:5,5,stroke-width:2px;

    class hb1,hb2,hb3 overlay;
    class gm1,gm2,gm3,gm4,gm5,gm6,gm7,gm8,gm9,gm10,gm11 routing;
    class kv1,kv2,kv3,kv4,kv5,kv6 kv;
    class sup1,sup2,sup3,sup4,sup5,sup6 supervision;

    %% Click Events for Overlay Builder
    click hb1 "https://github.com/dcos/lashup/blob/master/src/lashup_hyparview_membership.erl"
    click hb2 "https://github.com/dcos/lashup/blob/master/src/lashup_hyparview_ping_handler.erl"
    click hb3 "https://github.com/dcos/lashup/blob/master/src/lashup_hyparview_events.erl"

    %% Click Events for Routing & Failure Detection
    click gm1 "https://github.com/dcos/lashup/blob/master/src/lashup_gm.erl"
    click gm2 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_events.erl"
    click gm3 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_route.erl"
    click gm4 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_route_events.erl"
    click gm5 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_fanout.erl"
    click gm6 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_mc.erl"
    click gm7 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_mc_events.erl"
    click gm8 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_probe.erl"
    click gm9 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_sup.erl"
    click gm10 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_worker_sup.erl"
    click gm11 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_sync_worker.erl"

    %% Click Events for Distributed CRDT Store (KV)
    click kv1 "https://github.com/dcos/lashup/blob/master/src/lashup_kv.erl"
    click kv2 "https://github.com/dcos/lashup/blob/master/src/lashup_kv_aae_mgr.erl"
    click kv3 "https://github.com/dcos/lashup/blob/master/src/lashup_kv_sync_rx_fsm.erl"
    click kv4 "https://github.com/dcos/lashup/blob/master/src/lashup_kv_sync_tx_fsm.erl"
    click kv5 "https://github.com/dcos/lashup/blob/master/src/lashup_kv_sup.erl"
    click kv6 "https://github.com/dcos/lashup/blob/master/src/lashup_kv_aae_sup.erl"

    %% Click Events for Supervision & Infrastructure
    click sup1 "https://github.com/dcos/lashup/blob/master/src/lashup_core_sup.erl"
    click sup2 "https://github.com/dcos/lashup/blob/master/src/lashup_platform_sup.erl"
    click sup3 "https://github.com/dcos/lashup/blob/master/src/lashup_sup.erl"
    click sup4 "https://github.com/dcos/lashup/blob/master/src/lashup_gm_mc_sup.erl"
    click sup5 "https://github.com/dcos/lashup/blob/master/src/lashup_config.erl"
    click sup6 "https://github.com/dcos/lashup/blob/master/src/lashup_utils.erl"
```
