# terry-mao/bfs architecture

> Architecture diagram of the GitHub repository terry-mao/bfs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/terry-mao/bfs
- Repository: https://github.com/terry-mao/bfs

## Overview

##### Project type and purpose
- This is a distributed, full-stack small-file storage system (“bfs”), written in Go (core services) and Python (operations UI), inspired by Facebook Haystack.
- Core modules: directory (metadata & scheduling), store (physical file storage), pitchfork (node monitoring), proxy (bucket API & request routing), ops (Python backend UI).
- External dependencies: HBase (metadata persistence), ZooKeeper (coordination), Snowflake (unique key generation via gosnowflake).

##### File structure
- Top-level service directories:
- directory/ – metadata service code, tests, config
- store/ – storage node code, volumes, OS integration, config
- pitchfork/ – health-check & monitoring service
- proxy/ – client-facing API gateway & bucket management
- ops/ – Python Flask/Django-style UI for cluster operations
- libs/ – shared Go libraries (gohbase client, encoding, errors, memcache, meta, stat, uuid, time)
- Each service has config.go/.toml, HTTP API handlers, signal handling, unit tests, and main.go.

##### README for additional insights
- README describes a four-service cluster: directory, store, pitchfork, proxy, plus ops UI.
- It specifies each component’s responsibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend/API Layer
    subgraph "Frontend/API Layer"
        Client["Client/User"]:::external
        Proxy["Proxy Service\n(proxy/main.go)"]:::frontend
        OpsUI["Ops UI\n(ops/runserver.py)"]:::frontend
    end

    %% Metadata Layer
    subgraph "Metadata Layer"
        Directory["Directory Service\n(directory/main.go)"]:::metadata
        HBase["HBase Cluster"]:::database
        Snowflake["Snowflake ID Generator"]:::metadata
    end

    %% Storage Layer
    subgraph "Storage Layer"
        Store["Store Node Service\n(store/main.go)"]:::storage
    end

    %% Monitoring/Coordination Layer
    subgraph "Monitoring/Coordination Layer"
        Pitchfork["Pitchfork Service\n(pitchfork/main.go)"]:::monitor
        ZooKeeper["ZooKeeper Ensemble"]:::database
    end

    %% Interactions
    Client -->|"Upload/Download HTTP API"| Proxy
    Proxy -->|"BucketAPI()/Lookup()"| Directory
    Directory -->|"GenerateID()"| Snowflake
    Directory -->|"Read/Write Metadata"| HBase
    Directory -->|"Register/Discover"| ZooKeeper
    Directory -->|"Schedule Read/Write"| Store
    Proxy -->|"Data Transfer POST/GET"| Store
    Pitchfork -->|"Heartbeat/Stats"| Store
    Store -->|"Heartbeat Ack"| Pitchfork
    Pitchfork -->|"Service Registration"| ZooKeeper
    OpsUI -->|"Management APIs"| Directory
    OpsUI -->|"Management APIs"| Store
    OpsUI -->|"Management APIs"| Pitchfork

    %% Click Events - Directory Service
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/main.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/directory.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/http_api.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/dispatcher.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/conf/config.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/directory.toml"
    click HBase "https://github.com/terry-mao/bfs/blob/master/directory/hbase/file.go"
    click HBase "https://github.com/terry-mao/bfs/blob/master/directory/hbase/hbase_client.go"
    click HBase "https://github.com/terry-mao/bfs/blob/master/directory/hbase/needle.go"
    click Snowflake "https://github.com/terry-mao/bfs/blob/master/directory/snowflake/generate_key.go"
    click Snowflake "https://github.com/terry-mao/bfs/blob/master/directory/snowflake/gosnowflake.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/directory/zk/zk.go"

    %% Click Events - Store Service
    click Store "https://github.com/terry-mao/bfs/blob/master/store/main.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/store.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/http_api.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/conf/config.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/store.toml"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/needle/needle.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/needle/needles.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/needle/cache.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/block/supper_block.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/index/index.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/index/ring.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/volume/volume.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/os/file.go"
    click Store "https://github.com/terry-mao/bfs/blob/master/store/zk/zk.go"

    %% Click Events - Proxy Service
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/main.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/http_api.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/conf/config.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/proxy.toml"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/auth/auth.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/bfs/bfs.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/bucket/bucket.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/cache/cache.go"
    click Proxy "https://github.com/terry-mao/bfs/blob/master/proxy/service.go"

    %% Click Events - Pitchfork Service
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/main.go"
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/pitchfork.go"
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/conf/config.go"
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/pitchfork.toml"
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/signal.go"
    click Pitchfork "https://github.com/terry-mao/bfs/blob/master/pitchfork/zk/zk.go"

    %% Click Events - Ops UI
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/runserver.py"
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/setup.py"
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/commons/config.py"
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/commons/zk_client.py"
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/commons/store_client.py"
    click OpsUI "https://github.com/terry-mao/bfs/blob/master/ops/views/ops.py"

    %% Click Events - Shared Libraries
    click Directory "https://github.com/terry-mao/bfs/blob/master/libs/gohbase/client.go"
    click Directory "https://github.com/terry-mao/bfs/blob/master/libs/gohbase/zk/client.go"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/encoding/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/errors/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/memcache/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/meta/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/stat/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/time/"
    click Directory "https://github.com/terry-mao/bfs/tree/master/libs/uuid/"

    %% Styles
    classDef frontend fill:#b3e5fc,stroke:#0288d1,color:#000
    classDef metadata fill:#fff9c4,stroke:#f9a825,color:#000
    classDef storage fill:#c8e6c9,stroke:#2e7d32,color:#000
    classDef monitor fill:#ffcdd2,stroke:#c62828,color:#000
    classDef database fill:#e1bee7,stroke:#6a1b9a,color:#000
    classDef external fill:#eeeeee,stroke:#757575,color:#000
```
