How hpc/spindle works
Project type and purpose
• This is an HPC middleware tool (not a web or mobile app). It transparently intercepts dynamic library loads in large-scale MPI/Python jobs and implements a scalable broadcast to relieve shared‐FS pressure.
File structure
• Top‐level “src” is subdivided into:
– client: preloaded library code intercepting dlopen/open calls in user processes
– server: daemon code that coordinates library loads and metadata (auditserver, cache, comlib)
– fe (front‐end): code that runs on the login/head node to bootstrap and launch the back‐end under LaunchMON or scheduler plugins
– flux / slurm_plugin / cobo / biter: optional components integrating with Flux scheduler, SLURM, handshake/broadcast libraries, lightweight I/O server (biter)
• “include” holds public APIs, “utils” supplies shared helpers, “logging” implements debug/log facilities, “scripts” and autotools files handle build/deploy.
README for insights
• Spindle wraps an application launch command, intercepts dynamic loads, designates one process to read from the shared file‐system, then uses a tree‐based broadcast to all ranks.
• It works without special hardware or kernel patches, relies purely on userspace.
Main components and their interactions (to be depicted
a. External Scheduler / Launcher
b. Spindle Wrapper (spindle CLI)
– Parses args, selects “frontend” or “backend” mode.
c. Front‐end (spindle_fe) on head node
– Integrates with LaunchMON or scheduler plugin (slurm_plugin, flux-spindle)
– Spawns backend processes under LaunchMON’s control
d. Back‐end (spindle_be) running on each compute node
– Communicates with the front‐end over LaunchMON’s control channel
– Listens for “load library” requests via the client interpose layer
e. Client Interpose Library (ld‐audit + LD_PRELOAD)
– Linked into every MPI rank / Python process
– Intercepts dlopen/open/readlink/stat calls
– Queries a local “biterd” or audit‐server process to determine if a broadcast is needed
f. Audit Server / Biter Daemons
– auditserver: validates file metadata, prevents redundant loads
– biter: fast I/O server that can read file chunks from shared FS and supply them
g. Cobo (optional broadcast library)
– Provides scalable point‐to‐point or collective broadcast of file bytes over sockets or shared‐memory
h. Cache layers
– Server‐side caches paths and file‐attributes to mitigate repeated metadata ops
i. Technologies & protocols
– Unix domain sockets, TCP sockets, LaunchMON control channel, shared memory (shm), pipes (comlib), autotools build
Architectural patterns and design principles
• Client–Server: The client interposer talks to a local server process (biterd or auditserver).
• Tree‐based Broadcast: Cobo or custom binomial tree distributes library bits.
• Separation of Concerns:
– Front‐end handles job startup and scheduler integration
– Back‐end handles per‐node coordination
– Client library only intercepts loader calls and hands off to the back‐end
• Extensibility points: slurm_plugin, flux plugin, multiple handshake/backends under src/cobo