# halalcloud/kagami-storage architecture

> Architecture diagram of the GitHub repository halalcloud/kagami-storage, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/halalcloud/kagami-storage
- Repository: https://github.com/halalcloud/kagami-storage

## Overview

kagami-storage is a Haystack-inspired small-file storage system. The documented request path centers on proxying client storage requests, directory-based scheduling and metadata, and physical storage on store nodes; ops provides administrative provisioning, while pitchfork monitors store health. README descriptions establish these roles, but sampled source does not show the proxy-to-directory/store call chain, so those links are marked unverified. The graph also shows ops’ evidenced store-admin calls and ZooKeeper-backed cluster-state handling; other auxiliary utilities are omitted.

Main entry points: [`ops/commons/store_client.py`](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/store_client.py) (Store admin client), [`ops/commons/zk_client.py`](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/zk_client.py) (Cluster state client), [`store/volume/volume.go`](https://github.com/halalcloud/kagami-storage/blob/main/store/volume/volume.go) (Physical store), [`proxy/efs/efs.go`](https://github.com/halalcloud/kagami-storage/blob/main/proxy/efs/efs.go) (Storage proxy)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Storage services"]
  node_proxy["Storage proxy<br/>[efs.go]"]
  node_directory["Directory service<br/>[directory.go]"]
  node_metadata[("HBase metadata<br/>[hbase.go]")]
  node_store[("Physical store<br/>[volume.go]")]
  node_needle["Needle records<br/>[needle.go]"]
  node_bucket["Bucket management<br/>[bucket.go]"]
end

subgraph group_control["Cluster control"]
  node_pitchfork["Store monitor<br/>[pitchfork.go]"]
  node_zk[("ZooKeeper<br/>[zk.go]")]
  node_rebalance["Rebalance service<br/>[rebalance.go]"]
end

subgraph group_admin["Administration"]
  node_adminui["Ops management UI<br/>[ops.py]"]
  node_storeclient["Store admin client<br/>[store_client.py]"]
  node_zkclient["Cluster state client<br/>[zk_client.py]"]
end

subgraph group_opsinternals["Ops internals"]
  node_opsconfig["Ops configuration<br/>[config.py]"]
  node_opsstate["Ops state<br/>[global_var.py]"]
  node_opslogging["Ops logging<br/>[blogging.py]"]
end

node_client(("Storage client"))

node_client -->|"sends requests"| node_proxy
node_proxy -.->|"routes requests"| node_directory
node_proxy -.->|"proxies storage"| node_store
node_directory -->|"manages metadata"| node_metadata
node_pitchfork -->|"monitors health"| node_store
node_adminui -.->|"uses admin client"| node_storeclient
node_storeclient -->|"calls admin API"| node_store
node_zkclient -->|"reads cluster state"| node_zk
node_zkclient -->|"updates state"| node_opsstate
node_storeclient -->|"updates store stats"| node_opsstate
node_zkclient -->|"uses settings"| node_opsconfig
node_storeclient -->|"uses settings"| node_opsconfig
node_zkclient -->|"logs events"| node_opslogging
node_storeclient -->|"logs failures"| node_opslogging
node_store -.->|"stores records"| node_needle

click node_proxy "https://github.com/halalcloud/kagami-storage/blob/main/proxy/efs/efs.go"
click node_directory "https://github.com/halalcloud/kagami-storage/blob/main/directory/directory.go"
click node_metadata "https://github.com/halalcloud/kagami-storage/blob/main/directory/hbase/hbase.go"
click node_store "https://github.com/halalcloud/kagami-storage/blob/main/store/volume/volume.go"
click node_needle "https://github.com/halalcloud/kagami-storage/blob/main/store/needle/needle.go"
click node_bucket "https://github.com/halalcloud/kagami-storage/blob/main/proxy/bucket/bucket.go"
click node_pitchfork "https://github.com/halalcloud/kagami-storage/blob/main/pitchfork/pitchfork.go"
click node_zk "https://github.com/halalcloud/kagami-storage/blob/main/directory/zk/zk.go"
click node_rebalance "https://github.com/halalcloud/kagami-storage/blob/main/rebalance/rebalance.go"
click node_adminui "https://github.com/halalcloud/kagami-storage/blob/main/ops/views/ops.py"
click node_storeclient "https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/store_client.py"
click node_zkclient "https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/zk_client.py"
click node_opsconfig "https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/config.py"
click node_opsstate "https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/global_var.py"
click node_opslogging "https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/blogging.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_proxy,node_directory,node_metadata,node_store,node_needle,node_bucket toneBlue
class node_pitchfork,node_zk,node_rebalance,node_client toneAmber
class node_adminui,node_storeclient,node_zkclient toneMint
class node_opsconfig,node_opsstate,node_opslogging toneRose
```

## Components

### Storage services

- [Storage proxy](https://github.com/halalcloud/kagami-storage/blob/main/proxy/efs/efs.go): `proxy/efs/efs.go`
- [Directory service](https://github.com/halalcloud/kagami-storage/blob/main/directory/directory.go): `directory/directory.go`
- [HBase metadata](https://github.com/halalcloud/kagami-storage/blob/main/directory/hbase/hbase.go): `directory/hbase/hbase.go`
- [Physical store](https://github.com/halalcloud/kagami-storage/blob/main/store/volume/volume.go): `store/volume/volume.go`
- [Needle records](https://github.com/halalcloud/kagami-storage/blob/main/store/needle/needle.go): `store/needle/needle.go`
- [Bucket management](https://github.com/halalcloud/kagami-storage/blob/main/proxy/bucket/bucket.go): `proxy/bucket/bucket.go`

### Cluster control

- [Store monitor](https://github.com/halalcloud/kagami-storage/blob/main/pitchfork/pitchfork.go): `pitchfork/pitchfork.go`
- [ZooKeeper](https://github.com/halalcloud/kagami-storage/blob/main/directory/zk/zk.go): `directory/zk/zk.go`
- [Rebalance service](https://github.com/halalcloud/kagami-storage/blob/main/rebalance/rebalance.go): `rebalance/rebalance.go`

### Administration

- [Ops management UI](https://github.com/halalcloud/kagami-storage/blob/main/ops/views/ops.py): `ops/views/ops.py`
- [Store admin client](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/store_client.py): `ops/commons/store_client.py`
- [Cluster state client](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/zk_client.py): `ops/commons/zk_client.py`

### Ops internals

- [Ops configuration](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/config.py): `ops/commons/config.py`
- [Ops state](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/global_var.py): `ops/commons/global_var.py`
- [Ops logging](https://github.com/halalcloud/kagami-storage/blob/main/ops/commons/blogging.py): `ops/commons/blogging.py`

### Other components

- **Storage client**

## Connections

- Storage client → Storage proxy: sends requests
- Storage proxy → Directory service: routes requests
- Storage proxy → Physical store: proxies storage
- Directory service → HBase metadata: manages metadata
- Store monitor → Physical store: monitors health
- Ops management UI → Store admin client: uses admin client
- Store admin client → Physical store: calls admin API (evidence: `ops/commons/store_client.py`)
- Cluster state client → ZooKeeper: reads cluster state (evidence: `ops/commons/zk_client.py`)
- Cluster state client → Ops state: updates state (evidence: `ops/commons/zk_client.py`)
- Store admin client → Ops state: updates store stats (evidence: `ops/commons/store_client.py`)
- Cluster state client → Ops configuration: uses settings (evidence: `ops/commons/zk_client.py`)
- Store admin client → Ops configuration: uses settings (evidence: `ops/commons/store_client.py`)
- Cluster state client → Ops logging: logs events (evidence: `ops/commons/zk_client.py`)
- Store admin client → Ops logging: logs failures (evidence: `ops/commons/store_client.py`)
- Physical store → Needle records: stores records
