How guillaume-be/rust-bert works
Main Components
• Core Library (src/lib.rs): This is the heart of the project. It includes:
• Common components (in src/common): Modules for activations, dropout, embeddings, configuration, error handling, and resource management (local, remote, buffer).
• Model implementations (in src/models): Each transformer model (BERT, GPT2, ALBERT, etc.) is organized into its own directory with submodules for attention mechanisms, encoder/decoder layers, embeddings, and other model-specific logic.
• Pipelines (in src/pipelines): These combine the core models with task-specific logic. Different pipelines exist for tasks like question answering, sentiment analysis, translation, summarization, text generation, NER, etc.
• Examples & Benchmarks (examples, benches): These directories provide executable examples and benchmarks to demonstrate usage and measure performance.
• Tests (tests): Contains unit tests for individual model implementations and pipelines.
• Utilities (utils): Scripts and helper modules for operations like converting model weights.
External Dependencies & Interactions
• The project uses the tch-rs crate (Rust binding for Libtorch) for model execution using native or GPU-accelerated inference.
• Optionally, it includes ONNX support via bindings from the ort crate.
• It relies on rust-tokenizers for pre-processing and tokenization.
• Additional build and configuration files (Cargo.toml, build.rs) manage the Rust build process and dependencies.
Determining Relationships & Data Flow
• At the top level, the user’s code (or the examples) calls one of the ready-to-use pipeline APIs.
• The pipeline modules encapsulate task-specific logic, calling into the model components (from src/models) and leveraging shared common utility functions (from src/common).
• Model components then rely on lower-level tensor operations through tch-rs (or ONNX if the optional feature is enabled), which in turn interacts with the underlying C++ libraries (libtorch or onnxruntime).
• The utilities (such as the conversion script) interface with external pre-trained model weights to prepare or convert them to the required format.
Architectural Patterns & Design Principles
• Modularization: There is a clear separation between “common” utilities, model definitions, and task-specific pipelines.
• Layered design: The top-level APIs and pipelines abstract away the low-level details of model execution and pre-processing.
• Extensibility: With separate modules for each transformer model and pipeline, adding a new model architecture or pipeline is straightforward.
• Feature flags: Optional ONNX support is implemented as a feature flag, demonstrating the use of conditional compilation and configuration.