How aoboy/open_storage works
Storage Dataflow Stack implements storage fill, dispatch, repair, and pacing mechanisms, with a Python control plane coordinating live C data-plane components through shared memory. The main operational path is the CLI starting the service, which ticks controllers; WM profiles are published across SHM and consumed by the C proxy for origin dispatch. FLUID repair and fountain codecs, plus the ARC pacing sidecar, are major data-plane capabilities. The graph also shows metrics and the optional Rust LT arm. Relationships involving unsampled implementations are included only where the README documents them.
Main entry points: service.py Control service · wm_proxy.c Cache-fill proxy · sdf_cffi.py CFFI bridge · fluid_repair_main.c FLUID repair
Components 19
Control plane
- SDF CLI
storage-dataflow/python/sdf/cli/main.py - Control service
storage-dataflow/python/sdf/control/service.py - WM controller
storage-dataflow/python/sdf/control/wm_control.py - FLUID coordinator
storage-dataflow/python/sdf/control/fluid_coord.py - ARC deployment
storage-dataflow/python/sdf/control/arc_deploy.py - Python simulators
storage-dataflow/python/sdf/sim - Health reporting
storage-dataflow/python/sdf/control/health.py
Dispatch and origins
- Cache-fill proxy
storage-dataflow/src/c/whack_a_mole/wm_proxy.c - WM dispatch
storage-dataflow/include/sdf/wm.h - Origin drivers
storage-dataflow/src/c/origin
Repair and coding
- FLUID repair
storage-dataflow/src/c/fluid/fluid_repair_main.c - C fountain codecs
storage-dataflow/src/c/fountain - Rust LT arm
storage-dataflow/src/rust/sdf-fountain/src/lt.rs
ARC pacing
- ARC sidecar
storage-dataflow/src/c/arc/arc_sidecar_main.c - ARC scheduler
storage-dataflow/src/rust/sdf-arc/src/scheduler.rs
Shared runtime
- WM shared memory
storage-dataflow/python/sdf/common/shm.py - Metrics primitives
storage-dataflow/include/sdf/common/metrics.h - CFFI bridge
storage-dataflow/bindings/cffi/sdf_cffi.py
Other components
- Operator
Connections 22
- Operator → SDF CLI: runs
storage-dataflow/python/sdf/cli/main.py - SDF CLI → Control service: starts
storage-dataflow/python/sdf/cli/main.py - Control service → WM controller: ticks
storage-dataflow/python/sdf/control/service.py - Control service → ARC deployment: builds tickables
storage-dataflow/python/sdf/control/service.py - Control service → WM shared memory: attaches
README.md - Control service → Health reporting: writes health
README.md - WM controller → WM shared memory: publishes profile
storage-dataflow/python/sdf/control/wm_control.py - WM shared memory → Cache-fill proxy: shares telemetry
README.md - Cache-fill proxy → WM dispatch: dispatches units
README.md - Cache-fill proxy → Origin drivers: fetches ranges
README.md - FLUID repair → Origin drivers: fetches symbols
storage-dataflow/src/c/fluid/fluid_repair_main.c - FLUID repair → C fountain codecs: encodes and decodes
storage-dataflow/src/c/fluid/fluid_repair_main.c - CFFI bridge → Cache-fill proxy: bridges data plane
README.md - CFFI bridge → ARC sidecar: bridges data plane
README.md - CFFI bridge → FLUID repair: bridges data plane
README.md - CFFI bridge → C fountain codecs: bridges codecs
README.md - Rust LT arm → C fountain codecs: optional LT arm
README.md - ARC deployment → ARC sidecar: configures
README.md - ARC sidecar → ARC scheduler: runs schedule
README.md - ARC sidecar → Metrics primitives: records metrics
storage-dataflow/src/c/arc/arc_sidecar_main.c - Cache-fill proxy → WM shared memory: publishes telemetry
README.md - FLUID coordinator → FLUID repair: sets repair policy
README.md