How dropbox/rust-brotli-decompressor works
Project type and purpose
• This is a Rust library crate (“rust-brotli-decompressor”) whose core functionality is a direct, no-stdlib port of Google’s Brotli decompressor.
• It provides:
• A Rust-native API (std/no-std) for streaming and one-shot decompression.
• A C FFI layer (cdylib) matching the original C Brotli API.
• Command-line/test binaries, example code, fuzz tests, and integration tests.
File structure
Top-level:
• Cargo.toml, README.md, LICENSE, CI config (.travis.yml) – crate metadata and build pipeline.
• c/ – C FFI directory containing a Makefile and C example (main.c). Builds a cdylib for C linkage.
• src/ – library code, subdivided into:
– bin/ – multiple Rust binaries (CLI decompressor, tests harness, error tests).
– modules: bit_reader, context, decode, dictionary, ffi, huffman, memory, prefix, reader, state, transform, writer, plus helper files (io_wrappers.rs, brotli_alloc.rs).
– lib.rs – top-level entrypoint; re-exports public API.
• examples/ – standalone Rust example (decompress.rs).
• fuzz/ – Cargo project with fuzz targets that import the library and run fuzzing on decompression routines.
• testdata/ – corpus of sample files for tests and integration.
• c/Makefile – builds the C example and the cdylib.
Key architectural notes from README
• Dual build modes: with-std (default) vs. no-stdlib (feature-gated).
• FFI exports are feature-gated (“ffi-api”).
• Custom allocator support for kernel or no-syscall environments.
• Supports streaming (BrotliDecompressStream) and one-shot (BrotliDecoderDecompress).
Main components
A. Core Rust Decompressor Library
• Internal modules: bit_reader, huffman, dictionary, prefix, transform, decode, context, state, memory.
• Public API: Decompressor (io::Read), BrotliDecompress (stream copy), low-level BrotliDecompressStream.
• Feature flags: std, no-stdlib, ffi-api.
B. Rust CLI & Test Binaries
• src/bin/brotli-decompressor.rs – CLI entrypoint.
• src/bin/error_handling_tests.rs, integration_tests.rs, tests.rs – test harnesses using testdata.
C. Examples & Fuzzing
• examples/decompress.rs – plain-Rust usage demo.
• fuzz/fuzz_targets/*.rs – cargo-fuzz harness targets.
D. C FFI Layer
• c/ffi (in Rust src/ffi) – binds to core library.
• c/main.c – C example illustrating BrotliDecoderStream API.
• c/Makefile – compiles cdylib and example.
E. Testdata & CI
• testdata/ – decompression inputs for integration tests.
• .travis.yml – CI pipeline runs cargo test, fuzz, and C build.
Relationships and data flows
• Rust CLI binaries → call public API in core library → modules decode data → writer outputs decompressed bytes.
• examples/decompress.rs → uses the same public API.
• fuzz targets → consume public API internals to feed random inputs and catch panics.
• C example (main.c) → links against cdylib built from Rust FFI layer → calls BrotliDecoderStream (Rust implementation).
• Tests (src/bin/integration_tests.rs) → read from testdata/ → invoke library → compare with expected outputs.
Architectural patterns & design principles
• Layered modular decomposition: low-level bit readers → Huffman & LZ77 decode → transform + dictionary → high-level streaming API.
• Feature gating for optional components (ffi-api, no-stdlib).
• Zero-cost FFI design: cdylib exports match C symbols exactly.
• Separation of concerns: core library vs. CLI vs. FFI vs. test harnesses vs. examples vs. fuzzers.
Diagramming conventions
• From CLI/Examples/Fuzz → Core Library (calls).
• From Core Library → Testdata (reads).
• From C example → FFI Layer → Core Library.
• Library core = blue
• CLI/apps = green
• FFI/C = orange
• Tests/fuzz = purple
• Build Tools = gray