How facebookarchive/scribe works
Project Type and Purpose
Scribe is a distributed, real‑time log‐aggregation server written in C++ (src/) with a Python client library (lib/py/) and a Thrift interface (if/). Its purpose is to receive log messages from clients, route them into “buckets,” optionally write them to HDFS, and support dynamic reconfiguration.
File Structure
• Top‐level components:
• if/ – Thrift IDL defining the scribe RPC interface
• lib/py/ – Python client library
• src/ – C++ server code and plugins (stores, bucket updater, HDFS support)
• examples/ – usage examples and configs
• test/ – PHP and C++ tests
• Key modules in src/:
• scribe_server – main server loop using libevent
• conn_pool – connection‐pooling for outgoing writes
• store & store_queue – abstract store interface and queue
• dynamic_bucket_updater & network_dynamic_config – control plane for reconfiguring buckets at runtime
• HdfsFile – optional HDFS sink plugin
README for Stack & Patterns
• Dependencies: libevent (event loop), Boost, Thrift, fb303 (monitoring), optional Hadoop HDFS
• Architectural patterns:
• RPC service (Thrift)
• Plugin store architecture (abstract Store interface with HDFS or custom sinks)
• Control plane (dynamic configuration via network_dynamic_config)
• Client–server separation
Main Components & Interactions
A. Client Layer
– Python client (lib/py/scribe) or any Thrift client
– Uses scribe.thrift over a TCP socket
B. Ingestion / Frontend Layer
– scribe_server (C++): listens on configured ports via libevent
– Dispatches incoming messages into buckets based on hashing rules
C. Control Plane
– dynamic_bucket_updater: fetches bucket‐to‐server mapping from a central config (network_dynamic_config)
– network_dynamic_config: contact point for retrieving/updating cluster config
D. Storage Plugins
– store_queue: in‐memory queue buffering per‐bucket data
– Concrete Store implementations (e.g., HdfsFile for HDFS write‐through)
– Additional custom stores if loaded
E. External Services
– HDFS cluster (optional)
– Monitoring (fb303 RPC interface served by scribe_server)
Architectural Patterns & Design Principles
• Service façade: Thrift interface isolates clients from implementation
• Plugin architecture: Store interface lets you add new sinks (e.g., HDFS) without server changes
• Event‑driven I/O: libevent for high‐throughput, nonblocking RPC
• Dynamic config: control plane decoupled from data plane
Technologies & Frameworks to Call Out
• Apache Thrift (RPC IDL + codegen)
• libevent (I/O multiplexer)
• Boost C++ libraries
• Hadoop HDFS (optional sink)
• fb303 (runtime metrics)
• Python (client library packaging)