# dropbox/rust-brotli architecture

> Architecture diagram of the GitHub repository dropbox/rust-brotli, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dropbox/rust-brotli
- Repository: https://github.com/dropbox/rust-brotli

## Overview

##### Project Type and Purpose
- This is a systems-level Rust library and CLI tool that provides both a pure-Rust Brotli compression/decompression engine and a drop-in C-compatible FFI implementation.
- It also includes language wrappers (Go, Python) and CLI binaries (brotli, catbrotli, decompressors, validators).
- The primary purpose is to offer bitwise-identical Brotli functionality to Google’s C implementation, plus extended features (multithreading, custom allocators, concatenation, validation).

##### High-Level Architecture Components
┌─────────────────────────┐
│ 1. CLI Binaries (Rust)  │  ←— src/bin/*.rs
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 2. Pure-Rust Library    │  ←— src/lib.rs, src/enc/, src/concat/
│  • Encoder/Decoder      │    • enc/ (compression pipeline modules)
│  • FFI façade           │    • ffi/ (bindings to C and multicompress)
└─────────────────────────┘
▲                 ▲
│ calls           │ optionally calls
│                 │
┌───────────────┐   ┌───────────────┐
│ 3. C Implementation │←─│ 4. C FFI Bindings │
│    (native brotli.c)│   │  (c/ and c/* headers)│
└───────────────┘   └───────────────┘
│                      ▲
│                      │
│                      │
┌───────────────────┐       │
│ 5. Language Wrappers    │──┘
│    • Go (c/go/brotli)   │
│    • Python (c/py/)     │
└───────────────────┘

##### Relationships and Data Flows
- CLI Binaries → Pure-Rust Library: invoke compressor/decompressor APIs.
- Pure-Rust Library → C Implementation (optional): via FFI façade modules when the ffi-api feature is enabled, calling into c/target/…/libbrotli.so.
- Pure-Rust Library → Language Wrappers: the Go/Python code links against the FFI bindings to expose the same Brotli functions.
- Testdata/Examples: feed into CLI or library interfaces for integration tests and benchmarks.

##### Architectural Patterns and Principles
- Modular Pipeline: src/enc/ is subdivided into logical stages (block splitting, entropy encoding, backward references, writer).
- FFI Facade Pattern: src/ffi/ modules isolate unsafe C bindings behind a safe Rust API.
- Separation of Concerns:
- Core compression logic in enc/
- Public interface in src/lib.rs
- Binaries in src/bin/
- C build in c/
- Language-specific bindings in c/go/, c/py/
- Feature Flags: cargo features toggle FFI vs. pure-Rust engine, SIMD support, multithreading.

- “CLI Binaries (src/bin)”
- “Rust Core Library (src/lib + enc/ + concat/)”
- “C Brotli Engine (c/)”
- “FFI Facade (src/ffi + c headers)”
- “Go Wrapper (c/go/)”
- “Python Wrapper (c/py/)”
c. Connect:
- Rust Core ↔ FFI Facade (solid)
- FFI Facade → C Brotli (solid)
- Wrappers → FFI Facade (solid)
- Rust (edition 2018, Cargo)
- C (Makefile, libbrotli.so)
- Go modules, Python setuptools

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Project Container
    subgraph "rust-brotli Project"
        direction TB

        %% Testdata / Examples
        TestData["Testdata / Examples"]:::external

        %% CLI Layer
        subgraph "CLI Binaries"
            CLI["CLI Binaries\n(src/bin/)"]:::rustcli
        end

        %% Rust Core Library
        subgraph "Pure-Rust Core"
            RustAPI["Rust Core Library\n(src/lib.rs)"]:::rustcore
            Enc["Encoder/Decoder Modules\n(src/enc/)"]:::rustcore
            Concat["Concatenation Module\n(src/concat/)"]:::rustcore
        end

        %% FFI Facade
        subgraph "FFI Facade"
            FFI["FFI Facade\n(src/ffi/)"]:::ffi
            Multi["Multithread Tests\n(src/ffi/multicompress/)"]:::ffi
        end

        %% C FFI Bindings (Rust wrapper for C)
        subgraph "C FFI Bindings"
            CRustBind["Rust Bindings\n(c/src/lib.rs)"]:::ffi
        end

        %% C Brotli Engine
        subgraph "C Brotli Engine"
            CImpl["C Engine\n(c/brotli.c & c/brotli/)"]:::cmod
            Make["C Build & Headers\n(c/Makefile, arg.h, custom_alloc.h, vec_u8.h)"]:::cmod
        end

        %% Language Wrappers
        subgraph "Language Wrappers"
            Go["Go Wrapper\n(c/go/brotli/)"]:::wrapper
            GoEntr["Go Entrypoint & Tests\n(c/go/main.go,\ndata_test.go, interface_test.go)"]:::wrapper
            Py["Python Wrapper\n(c/py/)"]:::wrapper
        end
    end

    %% Connections
    TestData -->|"tests & benchmarks"| CLI
    TestData -->|"tests & benchmarks"| RustAPI

    CLI -->|"compression API call"| RustAPI

    RustAPI --> Enc
    RustAPI --> Concat
    RustAPI -->|"uses FFI Facade"| FFI
    RustAPI -.->|"optional FFI call"| CImpl

    FFI -->|"multithread helper"| Multi
    FFI -->|"FFI call"| CRustBind

    CRustBind -->|"links to"| CImpl

    Go -->|"links via FFI"| FFI
    GoEntr -->|"tests via FFI"| FFI
    Py -->|"links via FFI"| FFI

    %% Click Events
    click CLI "https://github.com/dropbox/rust-brotli/tree/master/src/bin/"
    click RustAPI "https://github.com/dropbox/rust-brotli/blob/master/src/lib.rs"
    click Enc "https://github.com/dropbox/rust-brotli/tree/master/src/enc/"
    click Concat "https://github.com/dropbox/rust-brotli/tree/master/src/concat/"
    click FFI "https://github.com/dropbox/rust-brotli/tree/master/src/ffi/"
    click Multi "https://github.com/dropbox/rust-brotli/tree/master/src/ffi/multicompress/"
    click CImpl "https://github.com/dropbox/rust-brotli/blob/master/c/brotli.c"
    click Make "https://github.com/dropbox/rust-brotli/tree/master/c/Makefile"
    click Make "https://github.com/dropbox/rust-brotli/blob/master/c/arg.h"
    click Make "https://github.com/dropbox/rust-brotli/blob/master/c/custom_alloc.h"
    click Make "https://github.com/dropbox/rust-brotli/blob/master/c/vec_u8.h"
    click CRustBind "https://github.com/dropbox/rust-brotli/blob/master/c/src/lib.rs"
    click Go "https://github.com/dropbox/rust-brotli/tree/master/c/go/brotli/"
    click GoEntr "https://github.com/dropbox/rust-brotli/blob/master/c/go/main.go"
    click GoEntr "https://github.com/dropbox/rust-brotli/blob/master/c/go/brotli/data_test.go"
    click GoEntr "https://github.com/dropbox/rust-brotli/blob/master/c/go/brotli/interface_test.go"
    click Py "https://github.com/dropbox/rust-brotli/blob/master/c/py/brotli.py"
    click Py "https://github.com/dropbox/rust-brotli/blob/master/c/py/__init__.py"

    %% Styles
    classDef rustcore fill:#ccffcc,stroke:#009900,color:#003300;
    classDef rustcli fill:#ddffdd,stroke:#009900,color:#003300;
    classDef cmod fill:#cce0ff,stroke:#0055cc,color:#003366;
    classDef ffi fill:#ffe0b3,stroke:#cc7500,color:#663d00;
    classDef wrapper fill:#e5ccff,stroke:#8000ff,color:#330066;
    classDef external fill:#f2f2f2,stroke:#999999,color:#333333,stroke-dasharray: 5 5;
```
