# aoboy/open_storage architecture

> Architecture diagram of the GitHub repository aoboy/open_storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aoboy/open_storage
- Repository: https://github.com/aoboy/open_storage

## Overview

Storage Dataflow Stack implements storage fill, dispatch, repair, and pacing mechanisms, with a Python control plane coordinating live C data-plane components through shared memory. The main operational path is the CLI starting the service, which ticks controllers; WM profiles are published across SHM and consumed by the C proxy for origin dispatch. FLUID repair and fountain codecs, plus the ARC pacing sidecar, are major data-plane capabilities. The graph also shows metrics and the optional Rust LT arm. Relationships involving unsampled implementations are included only where the README documents them.

Main entry points: [`storage-dataflow/python/sdf/control/service.py`](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/service.py) (Control service), [`storage-dataflow/src/c/whack_a_mole/wm_proxy.c`](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/whack_a_mole/wm_proxy.c) (Cache-fill proxy), [`storage-dataflow/bindings/cffi/sdf_cffi.py`](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/bindings/cffi/sdf_cffi.py) (CFFI bridge), [`storage-dataflow/src/c/fluid/fluid_repair_main.c`](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/fluid/fluid_repair_main.c) (FLUID repair)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Control plane"]
  node_cli["SDF CLI<br/>[main.py]"]
  node_service["Control service<br/>[service.py]"]
  node_wm_control["WM controller<br/>[wm_control.py]"]
  node_fluid_control["FLUID coordinator<br/>[fluid_coord.py]"]
  node_arc_deploy["ARC deployment<br/>[arc_deploy.py]"]
  node_simulators["Python simulators"]
  node_health["Health reporting<br/>[health.py]"]
end

subgraph group_dispatch["Dispatch and origins"]
  node_proxy["Cache-fill proxy<br/>[wm_proxy.c]"]
  node_wm_dispatch["WM dispatch<br/>[wm.h]"]
  node_origins["Origin drivers"]
end

subgraph group_repair["Repair and coding"]
  node_repair["FLUID repair"]
  node_fountain_c["C fountain codecs"]
  node_fountain_rust["Rust LT arm<br/>[lt.rs]"]
end

subgraph group_pacing["ARC pacing"]
  node_arc_sidecar["ARC sidecar<br/>[arc_sidecar_main.c]"]
  node_arc_scheduler["ARC scheduler<br/>[scheduler.rs]"]
end

subgraph group_shared["Shared runtime"]
  node_shm[("WM shared memory<br/>[shm.py]")]
  node_metrics["Metrics primitives<br/>[metrics.h]"]
  node_cffi["CFFI bridge<br/>[sdf_cffi.py]"]
end

node_operator(("Operator"))

node_operator -->|"runs"| node_cli
node_cli -->|"starts"| node_service
node_service -->|"ticks"| node_wm_control
node_service -->|"builds tickables"| node_arc_deploy
node_service -->|"attaches"| node_shm
node_service -->|"writes health"| node_health
node_wm_control -->|"publishes profile"| node_shm
node_shm -->|"shares telemetry"| node_proxy
node_proxy -->|"dispatches units"| node_wm_dispatch
node_proxy -->|"fetches ranges"| node_origins
node_repair -->|"fetches symbols"| node_origins
node_repair -->|"encodes and decodes"| node_fountain_c
node_cffi -->|"bridges data plane"| node_proxy
node_cffi -->|"bridges data plane"| node_arc_sidecar
node_cffi -->|"bridges data plane"| node_repair
node_cffi -->|"bridges codecs"| node_fountain_c
node_fountain_rust -.->|"optional LT arm"| node_fountain_c
node_arc_deploy -->|"configures"| node_arc_sidecar
node_arc_sidecar -.->|"runs schedule"| node_arc_scheduler
node_arc_sidecar -->|"records metrics"| node_metrics
node_proxy -->|"publishes telemetry"| node_shm
node_fluid_control -.->|"sets repair policy"| node_repair

click node_cli "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/cli/main.py"
click node_service "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/service.py"
click node_wm_control "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/wm_control.py"
click node_fluid_control "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/fluid_coord.py"
click node_arc_deploy "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/arc_deploy.py"
click node_simulators "https://github.com/aoboy/open_storage/tree/main/storage-dataflow/python/sdf/sim"
click node_shm "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/common/shm.py"
click node_proxy "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/whack_a_mole/wm_proxy.c"
click node_wm_dispatch "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/include/sdf/wm.h"
click node_origins "https://github.com/aoboy/open_storage/tree/main/storage-dataflow/src/c/origin"
click node_repair "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/fluid/fluid_repair_main.c"
click node_fountain_c "https://github.com/aoboy/open_storage/tree/main/storage-dataflow/src/c/fountain"
click node_fountain_rust "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/rust/sdf-fountain/src/lt.rs"
click node_arc_sidecar "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/arc/arc_sidecar_main.c"
click node_arc_scheduler "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/rust/sdf-arc/src/scheduler.rs"
click node_metrics "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/include/sdf/common/metrics.h"
click node_health "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/health.py"
click node_cffi "https://github.com/aoboy/open_storage/blob/main/storage-dataflow/bindings/cffi/sdf_cffi.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_service,node_wm_control,node_fluid_control,node_arc_deploy,node_simulators,node_health toneBlue
class node_proxy,node_wm_dispatch,node_origins toneAmber
class node_repair,node_fountain_c,node_fountain_rust toneMint
class node_arc_sidecar,node_arc_scheduler toneRose
class node_shm,node_metrics,node_cffi,node_operator toneIndigo
```

## Components

### Control plane

- [SDF CLI](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/cli/main.py): `storage-dataflow/python/sdf/cli/main.py`
- [Control service](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/service.py): `storage-dataflow/python/sdf/control/service.py`
- [WM controller](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/wm_control.py): `storage-dataflow/python/sdf/control/wm_control.py`
- [FLUID coordinator](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/fluid_coord.py): `storage-dataflow/python/sdf/control/fluid_coord.py`
- [ARC deployment](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/arc_deploy.py): `storage-dataflow/python/sdf/control/arc_deploy.py`
- [Python simulators](https://github.com/aoboy/open_storage/tree/main/storage-dataflow/python/sdf/sim): `storage-dataflow/python/sdf/sim`
- [Health reporting](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/control/health.py): `storage-dataflow/python/sdf/control/health.py`

### Dispatch and origins

- [Cache-fill proxy](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/whack_a_mole/wm_proxy.c): `storage-dataflow/src/c/whack_a_mole/wm_proxy.c`
- [WM dispatch](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/include/sdf/wm.h): `storage-dataflow/include/sdf/wm.h`
- [Origin drivers](https://github.com/aoboy/open_storage/tree/main/storage-dataflow/src/c/origin): `storage-dataflow/src/c/origin`

### Repair and coding

- [FLUID repair](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/fluid/fluid_repair_main.c): `storage-dataflow/src/c/fluid/fluid_repair_main.c`
- [C fountain codecs](https://github.com/aoboy/open_storage/tree/main/storage-dataflow/src/c/fountain): `storage-dataflow/src/c/fountain`
- [Rust LT arm](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/rust/sdf-fountain/src/lt.rs): `storage-dataflow/src/rust/sdf-fountain/src/lt.rs`

### ARC pacing

- [ARC sidecar](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/c/arc/arc_sidecar_main.c): `storage-dataflow/src/c/arc/arc_sidecar_main.c`
- [ARC scheduler](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/src/rust/sdf-arc/src/scheduler.rs): `storage-dataflow/src/rust/sdf-arc/src/scheduler.rs`

### Shared runtime

- [WM shared memory](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/python/sdf/common/shm.py): `storage-dataflow/python/sdf/common/shm.py`
- [Metrics primitives](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/include/sdf/common/metrics.h): `storage-dataflow/include/sdf/common/metrics.h`
- [CFFI bridge](https://github.com/aoboy/open_storage/blob/main/storage-dataflow/bindings/cffi/sdf_cffi.py): `storage-dataflow/bindings/cffi/sdf_cffi.py`

### Other components

- **Operator**

## Connections

- Operator → SDF CLI: runs (evidence: `storage-dataflow/python/sdf/cli/main.py`)
- SDF CLI → Control service: starts (evidence: `storage-dataflow/python/sdf/cli/main.py`)
- Control service → WM controller: ticks (evidence: `storage-dataflow/python/sdf/control/service.py`)
- Control service → ARC deployment: builds tickables (evidence: `storage-dataflow/python/sdf/control/service.py`)
- Control service → WM shared memory: attaches (evidence: `README.md`)
- Control service → Health reporting: writes health (evidence: `README.md`)
- WM controller → WM shared memory: publishes profile (evidence: `storage-dataflow/python/sdf/control/wm_control.py`)
- WM shared memory → Cache-fill proxy: shares telemetry (evidence: `README.md`)
- Cache-fill proxy → WM dispatch: dispatches units (evidence: `README.md`)
- Cache-fill proxy → Origin drivers: fetches ranges (evidence: `README.md`)
- FLUID repair → Origin drivers: fetches symbols (evidence: `storage-dataflow/src/c/fluid/fluid_repair_main.c`)
- FLUID repair → C fountain codecs: encodes and decodes (evidence: `storage-dataflow/src/c/fluid/fluid_repair_main.c`)
- CFFI bridge → Cache-fill proxy: bridges data plane (evidence: `README.md`)
- CFFI bridge → ARC sidecar: bridges data plane (evidence: `README.md`)
- CFFI bridge → FLUID repair: bridges data plane (evidence: `README.md`)
- CFFI bridge → C fountain codecs: bridges codecs (evidence: `README.md`)
- Rust LT arm → C fountain codecs: optional LT arm (evidence: `README.md`)
- ARC deployment → ARC sidecar: configures (evidence: `README.md`)
- ARC sidecar → ARC scheduler: runs schedule (evidence: `README.md`)
- ARC sidecar → Metrics primitives: records metrics (evidence: `storage-dataflow/src/c/arc/arc_sidecar_main.c`)
- Cache-fill proxy → WM shared memory: publishes telemetry (evidence: `README.md`)
- FLUID coordinator → FLUID repair: sets repair policy (evidence: `README.md`)
