# hpc/spindle architecture

> Architecture diagram of the GitHub repository hpc/spindle, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hpc/spindle
- Repository: https://github.com/hpc/spindle

## Overview

##### Project type and purpose
- This is an HPC middleware tool (not a web or mobile app). It transparently intercepts dynamic library loads in large-scale MPI/Python jobs and implements a scalable broadcast to relieve shared‐FS pressure.

##### File structure
- Top‐level “src” is subdivided into:
– client: preloaded library code intercepting dlopen/open calls in user processes
– server: daemon code that coordinates library loads and metadata (auditserver, cache, comlib)
– fe (front‐end): code that runs on the login/head node to bootstrap and launch the back‐end under LaunchMON or scheduler plugins
– flux / slurm_plugin / cobo / biter: optional components integrating with Flux scheduler, SLURM, handshake/broadcast libraries, lightweight I/O server (biter)
- “include” holds public APIs, “utils” supplies shared helpers, “logging” implements debug/log facilities, “scripts” and autotools files handle build/deploy.

##### README for insights
- Spindle wraps an application launch command, intercepts dynamic loads, designates one process to read from the shared file‐system, then uses a tree‐based broadcast to all ranks.
- It works without special hardware or kernel patches, relies purely on userspace.

##### Main components and their interactions (to be depicted
a. External Scheduler / Launcher
b. Spindle Wrapper (spindle CLI)
– Parses args, selects “frontend” or “backend” mode.
c. Front‐end (spindle_fe) on head node
– Integrates with LaunchMON or scheduler plugin (slurm_plugin, flux-spindle)
– Spawns backend processes under LaunchMON’s control
d. Back‐end (spindle_be) running on each compute node
– Communicates with the front‐end over LaunchMON’s control channel
– Listens for “load library” requests via the client interpose layer
e. Client Interpose Library (ld‐audit + LD_PRELOAD)
– Linked into every MPI rank / Python process
– Intercepts dlopen/open/readlink/stat calls
– Queries a local “biterd” or audit‐server process to determine if a broadcast is needed
f. Audit Server / Biter Daemons
– auditserver: validates file metadata, prevents redundant loads
– biter: fast I/O server that can read file chunks from shared FS and supply them
g. Cobo (optional broadcast library)
– Provides scalable point‐to‐point or collective broadcast of file bytes over sockets or shared‐memory
h. Cache layers
– Server‐side caches paths and file‐attributes to mitigate repeated metadata ops
i. Technologies & protocols
– Unix domain sockets, TCP sockets, LaunchMON control channel, shared memory (shm), pipes (comlib), autotools build

##### Architectural patterns and design principles
- Client–Server: The client interposer talks to a local server process (biterd or auditserver).
- Tree‐based Broadcast: Cobo or custom binomial tree distributes library bits.
- Separation of Concerns:
– Front‐end handles job startup and scheduler integration
– Back‐end handles per‐node coordination
– Client library only intercepts loader calls and hands off to the back‐end
- Extensibility points: slurm_plugin, flux plugin, multiple handshake/backends under src/cobo

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Scheduler / Launcher
    subgraph "External Scheduler / Launcher"
        SLURM["SLURM (sbatch)"]:::scheduler
        FLUX["Flux"]:::scheduler
        LAUNCHMON["LaunchMON"]:::scheduler
    end

    %% Spindle Wrapper
    subgraph "Spindle Wrapper"
        WrapperHeader["spindle_launch.h"]:::spindle
        WrapperMain["spindle_fe_main.cc"]:::spindle
    end

    %% Spindle Front-end & Back-end
    FE["Front-end (spindle_fe)"]:::spindle
    BE["Back-end (spindle_be)"]:::spindle

    %% Client & Daemons & Broadcast
    ClientLib["Client Interpose Library"]:::client
    AuditServer["Audit Server Daemon"]:::daemon
    Biter["Biter Daemon"]:::daemon
    Cobo["Cobo Broadcast Library"]:::broadcast

    %% External FS
    FS["Shared File System"]:::external

    %% Connections
    SLURM -->|job-launch| WrapperMain
    FLUX -->|job-launch| WrapperMain
    LAUNCHMON -->|job-launch| WrapperMain

    WrapperMain -->|bootstrap| FE
    FE -->|LaunchMON RPC| BE

    BE -->|Unix socket| ClientLib
    ClientLib -->|load request| AuditServer
    AuditServer -->|file fetch| FS
    FS -->|file chunks| AuditServer

    AuditServer -->|broadcast via Cobo| Cobo
    Cobo -->|distribute| ClientLib

    AuditServer -->|optional fast I/O| Biter
    Biter -->|read chunks| FS

    %% Click Events
    click SLURM "https://github.com/hpc/spindle/tree/devel/src/slurm_plugin/"
    click FLUX "https://github.com/hpc/spindle/tree/devel/src/flux/"
    click LAUNCHMON "https://github.com/hpc/spindle/tree/devel/src/fe/launchmon/"
    click WrapperHeader "https://github.com/hpc/spindle/blob/devel/include/spindle_launch.h"
    click WrapperMain "https://github.com/hpc/spindle/blob/devel/src/fe/startup/spindle_fe_main.cc"
    click FE "https://github.com/hpc/spindle/tree/devel/src/fe/"
    click BE "https://github.com/hpc/spindle/tree/devel/src/server/startup/"
    click ClientLib "https://github.com/hpc/spindle/tree/devel/src/client/client/"
    click AuditServer "https://github.com/hpc/spindle/tree/devel/src/server/auditserver/"
    click Biter "https://github.com/hpc/spindle/tree/devel/src/biter/"
    click Cobo "https://github.com/hpc/spindle/tree/devel/src/cobo/"

    %% Styles
    classDef scheduler fill:#a2fca2,stroke:#333,stroke-width:1px
    classDef spindle fill:#a2d2fc,stroke:#333,stroke-width:1px
    classDef client fill:#fcbf49,stroke:#333,stroke-width:1px
    classDef daemon fill:#fee440,stroke:#333,stroke-width:1px
    classDef broadcast fill:#bdb2ff,stroke:#333,stroke-width:1px
    classDef external fill:#dddddd,stroke:#333,stroke-width:1px
```
