How terry-mao/bfs works
Project type and purpose
• This is a distributed, full-stack small-file storage system (“bfs”), written in Go (core services) and Python (operations UI), inspired by Facebook Haystack.
• Core modules: directory (metadata & scheduling), store (physical file storage), pitchfork (node monitoring), proxy (bucket API & request routing), ops (Python backend UI).
• External dependencies: HBase (metadata persistence), ZooKeeper (coordination), Snowflake (unique key generation via gosnowflake).
File structure
• Top-level service directories:
• directory/ – metadata service code, tests, config
• store/ – storage node code, volumes, OS integration, config
• pitchfork/ – health-check & monitoring service
• proxy/ – client-facing API gateway & bucket management
• ops/ – Python Flask/Django-style UI for cluster operations
• libs/ – shared Go libraries (gohbase client, encoding, errors, memcache, meta, stat, uuid, time)
• Each service has config.go/.toml, HTTP API handlers, signal handling, unit tests, and main.go.
README for additional insights
• README describes a four-service cluster: directory, store, pitchfork, proxy, plus ops UI.
• It specifies each component’s responsibility.