# ryanbinxu/archpipe architecture

> Architecture diagram of the GitHub repository ryanbinxu/archpipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ryanbinxu/archpipe
- Repository: https://github.com/ryanbinxu/archpipe

## Overview

Archpipe is a C prototype for erasure-coded archival over asymmetric networks. Its apparent workflow is to initialize block placement and stripe layout, schedule encoding pipelines, issue work to data nodes, and perform encoding or recovery there. The excerpts show coordinator scheduling and command transmission, and data-node pipeline and central-recovery code. The README is only a one-line product description, so detailed capabilities and some runtime links are based on source references; unsampled dispatch paths are not asserted.

Main entry points: [`coordinator.c`](https://github.com/ryanbinxu/archpipe/blob/main/coordinator.c) (Coordinator), [`datanode.c`](https://github.com/ryanbinxu/archpipe/blob/main/datanode.c) (Data-node service), [`placement.c`](https://github.com/ryanbinxu/archpipe/blob/main/placement.c) (Block placement), [`stripecons.c`](https://github.com/ryanbinxu/archpipe/blob/main/stripecons.c) (Stripe construction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Coordination"]
  node_coordinator["Coordinator<br/>[coordinator.c]"]
  node_schedule_variant["Timing coordinator"]
  node_pipeline_schedule["Pipeline scheduling<br/>[coordinator.c]"]
end

subgraph group_layout["Data Layout"]
  node_placement["Block placement<br/>[placement.c]"]
  node_stripe_layout["Stripe construction<br/>[stripecons.c]"]
end

subgraph group_nodes["Data Nodes"]
  node_data_node["Data-node service<br/>[datanode.c]"]
  node_pipe_encode["Pipeline encoding<br/>[datanode.c]"]
  node_central_recovery["Central recovery<br/>[datanode.c]"]
  node_parity_transfer["Parity transfer<br/>[datanode.c]"]
end

subgraph group_runtime["Runtime Support"]
  node_gf_arithmetic["Galois arithmetic<br/>[galois.c]"]
  node_transport_protocol["Commands and transport<br/>[common.c]"]
  node_configuration["Runtime configuration<br/>[config.h]"]
  node_array_list["Array list<br/>[arraylist.c]"]
  node_cuckoo_hash["Cuckoo hash<br/>[cuckoo.c]"]
  node_hash_functions["Hash functions<br/>[hash.c]"]
end

node_operator(("Archival operator"))

node_operator -.->|"initiates archival"| node_coordinator
node_coordinator -->|"uses placement"| node_placement
node_coordinator -->|"constructs stripes"| node_stripe_layout
node_coordinator -->|"selects schedule"| node_pipeline_schedule
node_schedule_variant -->|"uses placement"| node_placement
node_schedule_variant -->|"constructs stripes"| node_stripe_layout
node_coordinator -->|"sends commands"| node_transport_protocol
node_coordinator -->|"transmits parity command"| node_data_node
node_data_node -.->|"dispatches work"| node_pipe_encode
node_data_node -.->|"dispatches work"| node_central_recovery
node_data_node -.->|"dispatches work"| node_parity_transfer
node_pipe_encode -->|"uses coding routines"| node_gf_arithmetic
node_central_recovery -.->|"uses coding routines"| node_gf_arithmetic
node_coordinator -->|"uses collections"| node_array_list
node_coordinator -->|"looks up node IPs"| node_cuckoo_hash
node_placement -->|"uses collections"| node_array_list
node_placement -->|"uses maps"| node_cuckoo_hash
node_stripe_layout -->|"uses collections"| node_array_list
node_stripe_layout -->|"uses maps"| node_cuckoo_hash
node_cuckoo_hash -->|"uses hashing"| node_hash_functions
node_coordinator -->|"reads settings"| node_configuration
node_data_node -->|"uses command protocol"| node_transport_protocol

click node_coordinator "https://github.com/ryanbinxu/archpipe/blob/main/coordinator.c"
click node_schedule_variant "https://github.com/ryanbinxu/archpipe/blob/main/coordinator4schedule-time.c"
click node_pipeline_schedule "https://github.com/ryanbinxu/archpipe/blob/main/coordinator.c"
click node_placement "https://github.com/ryanbinxu/archpipe/blob/main/placement.c"
click node_stripe_layout "https://github.com/ryanbinxu/archpipe/blob/main/stripecons.c"
click node_data_node "https://github.com/ryanbinxu/archpipe/blob/main/datanode.c"
click node_pipe_encode "https://github.com/ryanbinxu/archpipe/blob/main/datanode.c"
click node_central_recovery "https://github.com/ryanbinxu/archpipe/blob/main/datanode.c"
click node_parity_transfer "https://github.com/ryanbinxu/archpipe/blob/main/datanode.c"
click node_gf_arithmetic "https://github.com/ryanbinxu/archpipe/blob/main/galois.c"
click node_transport_protocol "https://github.com/ryanbinxu/archpipe/blob/main/common.c"
click node_configuration "https://github.com/ryanbinxu/archpipe/blob/main/config.h"
click node_array_list "https://github.com/ryanbinxu/archpipe/blob/main/arraylist.c"
click node_cuckoo_hash "https://github.com/ryanbinxu/archpipe/blob/main/cuckoo.c"
click node_hash_functions "https://github.com/ryanbinxu/archpipe/blob/main/hash.c"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_coordinator,node_schedule_variant,node_pipeline_schedule toneBlue
class node_placement,node_stripe_layout toneAmber
class node_data_node,node_pipe_encode,node_central_recovery,node_parity_transfer toneMint
class node_gf_arithmetic,node_transport_protocol,node_configuration,node_array_list,node_cuckoo_hash,node_hash_functions toneRose
class node_operator toneIndigo
```

## Components

### Coordination

- [Coordinator](https://github.com/ryanbinxu/archpipe/blob/main/coordinator.c): `coordinator.c`
- [Timing coordinator](https://github.com/ryanbinxu/archpipe/blob/main/coordinator4schedule-time.c): `coordinator4schedule-time.c`
- [Pipeline scheduling](https://github.com/ryanbinxu/archpipe/blob/main/coordinator.c): `coordinator.c`

### Data Layout

- [Block placement](https://github.com/ryanbinxu/archpipe/blob/main/placement.c): `placement.c`
- [Stripe construction](https://github.com/ryanbinxu/archpipe/blob/main/stripecons.c): `stripecons.c`

### Data Nodes

- [Data-node service](https://github.com/ryanbinxu/archpipe/blob/main/datanode.c): `datanode.c`
- [Pipeline encoding](https://github.com/ryanbinxu/archpipe/blob/main/datanode.c): `datanode.c`
- [Central recovery](https://github.com/ryanbinxu/archpipe/blob/main/datanode.c): `datanode.c`
- [Parity transfer](https://github.com/ryanbinxu/archpipe/blob/main/datanode.c): `datanode.c`

### Runtime Support

- [Galois arithmetic](https://github.com/ryanbinxu/archpipe/blob/main/galois.c): `galois.c`
- [Commands and transport](https://github.com/ryanbinxu/archpipe/blob/main/common.c): `common.c`
- [Runtime configuration](https://github.com/ryanbinxu/archpipe/blob/main/config.h): `config.h`
- [Array list](https://github.com/ryanbinxu/archpipe/blob/main/arraylist.c): `arraylist.c`
- [Cuckoo hash](https://github.com/ryanbinxu/archpipe/blob/main/cuckoo.c): `cuckoo.c`
- [Hash functions](https://github.com/ryanbinxu/archpipe/blob/main/hash.c): `hash.c`

### Other components

- **Archival operator**

## Connections

- Archival operator → Coordinator: initiates archival (evidence: `README.md`)
- Coordinator → Block placement: uses placement (evidence: `coordinator.c`)
- Coordinator → Stripe construction: constructs stripes (evidence: `coordinator.c`)
- Coordinator → Pipeline scheduling: selects schedule (evidence: `coordinator.c`)
- Timing coordinator → Block placement: uses placement (evidence: `coordinator4schedule-time.c`)
- Timing coordinator → Stripe construction: constructs stripes (evidence: `coordinator4schedule-time.c`)
- Coordinator → Commands and transport: sends commands (evidence: `coordinator.c`)
- Coordinator → Data-node service: transmits parity command (evidence: `coordinator.c`)
- Data-node service → Pipeline encoding: dispatches work (evidence: `datanode.c`)
- Data-node service → Central recovery: dispatches work (evidence: `datanode.c`)
- Data-node service → Parity transfer: dispatches work (evidence: `datanode.c`)
- Pipeline encoding → Galois arithmetic: uses coding routines (evidence: `datanode.c`)
- Central recovery → Galois arithmetic: uses coding routines
- Coordinator → Array list: uses collections (evidence: `coordinator.c`)
- Coordinator → Cuckoo hash: looks up node IPs (evidence: `coordinator.c`)
- Block placement → Array list: uses collections (evidence: `placement.c`)
- Block placement → Cuckoo hash: uses maps (evidence: `placement.c`)
- Stripe construction → Array list: uses collections (evidence: `stripecons.c`)
- Stripe construction → Cuckoo hash: uses maps (evidence: `stripecons.c`)
- Cuckoo hash → Hash functions: uses hashing (evidence: `cuckoo.c`)
- Coordinator → Runtime configuration: reads settings (evidence: `coordinator.c`)
- Data-node service → Commands and transport: uses command protocol (evidence: `datanode.c`)