How dropbox/rust-brotli works
Project Type and Purpose
• This is a systems-level Rust library and CLI tool that provides both a pure-Rust Brotli compression/decompression engine and a drop-in C-compatible FFI implementation.
• It also includes language wrappers (Go, Python) and CLI binaries (brotli, catbrotli, decompressors, validators).
• The primary purpose is to offer bitwise-identical Brotli functionality to Google’s C implementation, plus extended features (multithreading, custom allocators, concatenation, validation).
High-Level Architecture Components
┌─────────────────────────┐
│ 1. CLI Binaries (Rust) │ ←— src/bin/*.rs
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 2. Pure-Rust Library │ ←— src/lib.rs, src/enc/, src/concat/
│ • Encoder/Decoder │ • enc/ (compression pipeline modules)
│ • FFI façade │ • ffi/ (bindings to C and multicompress)
└─────────────────────────┘
▲ ▲
│ calls │ optionally calls
│ │
┌───────────────┐ ┌───────────────┐
│ 3. C Implementation │←─│ 4. C FFI Bindings │
│ (native brotli.c)│ │ (c/ and c/* headers)│
└───────────────┘ └───────────────┘
│ ▲
│ │
│ │
┌───────────────────┐ │
│ 5. Language Wrappers │──┘
│ • Go (c/go/brotli) │
│ • Python (c/py/) │
└───────────────────┘
Relationships and Data Flows
• CLI Binaries → Pure-Rust Library: invoke compressor/decompressor APIs.
• Pure-Rust Library → C Implementation (optional): via FFI façade modules when the ffi-api feature is enabled, calling into c/target/…/libbrotli.so.
• Pure-Rust Library → Language Wrappers: the Go/Python code links against the FFI bindings to expose the same Brotli functions.
• Testdata/Examples: feed into CLI or library interfaces for integration tests and benchmarks.
Architectural Patterns and Principles
• Modular Pipeline: src/enc/ is subdivided into logical stages (block splitting, entropy encoding, backward references, writer).
• FFI Facade Pattern: src/ffi/ modules isolate unsafe C bindings behind a safe Rust API.
• Separation of Concerns:
• Core compression logic in enc/
• Public interface in src/lib.rs
• Binaries in src/bin/
• C build in c/
• Language-specific bindings in c/go/, c/py/
• Feature Flags: cargo features toggle FFI vs. pure-Rust engine, SIMD support, multithreading.
• “CLI Binaries (src/bin)”
• “Rust Core Library (src/lib + enc/ + concat/)”
• “C Brotli Engine (c/)”
• “FFI Facade (src/ffi + c headers)”
• “Go Wrapper (c/go/)”
• “Python Wrapper (c/py/)”
c. Connect:
• Rust Core ↔ FFI Facade (solid)
• FFI Facade → C Brotli (solid)
• Wrappers → FFI Facade (solid)
• Rust (edition 2018, Cargo)
• C (Makefile, libbrotli.so)
• Go modules, Python setuptools