# facebookarchive/scribe architecture

> Architecture diagram of the GitHub repository facebookarchive/scribe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookarchive/scribe
- Repository: https://github.com/facebookarchive/scribe

## Overview

##### Project Type and Purpose
Scribe is a distributed, real‑time log‐aggregation server written in C++ (src/) with a Python client library (lib/py/) and a Thrift interface (if/). Its purpose is to receive log messages from clients, route them into “buckets,” optionally write them to HDFS, and support dynamic reconfiguration.

##### File Structure
- Top‐level components:
- if/ – Thrift IDL defining the scribe RPC interface
- lib/py/ – Python client library
- src/ – C++ server code and plugins (stores, bucket updater, HDFS support)
- examples/ – usage examples and configs
- test/ – PHP and C++ tests
- Key modules in src/:
- scribe_server – main server loop using libevent
- conn_pool – connection‐pooling for outgoing writes
- store & store_queue – abstract store interface and queue
- dynamic_bucket_updater & network_dynamic_config – control plane for reconfiguring buckets at runtime
- HdfsFile – optional HDFS sink plugin

##### README for Stack & Patterns
- Dependencies: libevent (event loop), Boost, Thrift, fb303 (monitoring), optional Hadoop HDFS
- Architectural patterns:
- RPC service (Thrift)
- Plugin store architecture (abstract Store interface with HDFS or custom sinks)
- Control plane (dynamic configuration via network_dynamic_config)
- Client–server separation

##### Main Components & Interactions
A. Client Layer
– Python client (lib/py/scribe) or any Thrift client
– Uses scribe.thrift over a TCP socket
B. Ingestion / Frontend Layer
– scribe_server (C++): listens on configured ports via libevent
– Dispatches incoming messages into buckets based on hashing rules
C. Control Plane
– dynamic_bucket_updater: fetches bucket‐to‐server mapping from a central config (network_dynamic_config)
– network_dynamic_config: contact point for retrieving/updating cluster config
D. Storage Plugins
– store_queue: in‐memory queue buffering per‐bucket data
– Concrete Store implementations (e.g., HdfsFile for HDFS write‐through)
– Additional custom stores if loaded
E. External Services
– HDFS cluster (optional)
– Monitoring (fb303 RPC interface served by scribe_server)

##### Architectural Patterns & Design Principles
- Service façade: Thrift interface isolates clients from implementation
- Plugin architecture: Store interface lets you add new sinks (e.g., HDFS) without server changes
- Event‑driven I/O: libevent for high‐throughput, nonblocking RPC
- Dynamic config: control plane decoupled from data plane

##### Technologies & Frameworks to Call Out
- Apache Thrift (RPC IDL + codegen)
- libevent (I/O multiplexer)
- Boost C++ libraries
- Hadoop HDFS (optional sink)
- fb303 (runtime metrics)
- Python (client library packaging)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        PC1["__init__.py"]:::client
        PC2["constants.py"]:::client
        PC3["scribe.py"]:::client
        PC4["ttypes.py"]:::client
        PC5["scribe-remote"]:::client
        PC6["setup.py"]:::client
        Thrift1["scribe.thrift"]:::rpc
        Thrift2["bucketupdater.thrift"]:::rpc
    end

    %% Scribe Frontend Server Cluster
    subgraph "Scribe Frontend Server"
        direction TB
        SS["scribe_server.cpp/.h"]:::frontend
        CP["conn_pool.cpp/.h"]:::data
        SA["store.cpp/.h"]:::data
        SQ["store_queue.cpp/.h"]:::data
        FS["file.cpp/.h"]:::data
        HS["HdfsFile.cpp/.h"]:::data
        DS["dynamic_bucket_updater.cpp/.h"]:::control
        NDC["network_dynamic_config.cpp/.h"]:::control
        Conf1["conf.cpp/.h"]:::config
        Conf2["env_default.cpp/.h"]:::config
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        HDFS["HDFS Cluster"]:::external
        FB303["fb303 Monitoring"]:::external
    end

    %% Connections
    PC3 -->|Thrift/TCP RPC| SS
    Thrift1 -->|IDL| SS
    Thrift2 -->|IDL| DS

    SS -->|enqueue logs| SQ
    SQ -->|buffered logs| SA
    SA -->|writes to| FS
    SA -->|writes to| HS
    HS -->|HDFS RPC| HDFS

    SS -->|metrics| FB303

    SS <-->|bucket updates| DS
    DS <-->|fetch config| NDC

    %% Click Events for component mapping
    click Thrift1 "https://github.com/facebookarchive/scribe/blob/master/if/scribe.thrift"
    click Thrift2 "https://github.com/facebookarchive/scribe/blob/master/if/bucketupdater.thrift"
    click PC1 "https://github.com/facebookarchive/scribe/blob/master/lib/py/scribe/__init__.py"
    click PC2 "https://github.com/facebookarchive/scribe/blob/master/lib/py/scribe/constants.py"
    click PC3 "https://github.com/facebookarchive/scribe/blob/master/lib/py/scribe/scribe.py"
    click PC4 "https://github.com/facebookarchive/scribe/blob/master/lib/py/scribe/ttypes.py"
    click PC5 "https://github.com/facebookarchive/scribe/tree/master/lib/py/scribe/scribe-remote"
    click PC6 "https://github.com/facebookarchive/scribe/blob/master/lib/py/setup.py"
    click SS "https://github.com/facebookarchive/scribe/blob/master/src/scribe_server.cpp"
    click CP "https://github.com/facebookarchive/scribe/blob/master/src/conn_pool.cpp"
    click SA "https://github.com/facebookarchive/scribe/blob/master/src/store.cpp"
    click SQ "https://github.com/facebookarchive/scribe/blob/master/src/store_queue.cpp"
    click FS "https://github.com/facebookarchive/scribe/blob/master/src/file.cpp"
    click HS "https://github.com/facebookarchive/scribe/blob/master/src/HdfsFile.cpp"
    click DS "https://github.com/facebookarchive/scribe/blob/master/src/dynamic_bucket_updater.cpp"
    click NDC "https://github.com/facebookarchive/scribe/blob/master/src/network_dynamic_config.cpp"
    click Conf1 "https://github.com/facebookarchive/scribe/blob/master/src/conf.cpp"
    click Conf2 "https://github.com/facebookarchive/scribe/blob/master/src/env_default.cpp"

    %% Styles
    classDef client fill:#D0E1FF,stroke:#3465A4,color:#000;
    classDef rpc fill:#BBDFF2,stroke:#2A7F9E,color:#000;
    classDef frontend fill:#E0F7FA,stroke:#00796B,color:#000;
    classDef data fill:#FFE0B2,stroke:#FB8C00,color:#000;
    classDef control fill:#E8F5E9,stroke:#388E3C,color:#000;
    classDef config fill:#FFF3E0,stroke:#FFB300,color:#000;
    classDef external fill:#ECEFF1,stroke:#607D8B,color:#000;
```
