How bojone/bert4keras works
Project type and purpose
• This is an open‑source Python library (“bert4keras”) that re‑implements BERT‑style transformer models on top of Keras/TF‑Keras.
• It provides building blocks (tokenizers, backend wrappers, layer definitions, model constructors, optimizers, utility snippets), example scripts for common tasks (fine‑tuning, language modeling, sequence labeling, web serving), and an end‑to‑end pretraining pipeline (with TPU/multi‑GPU support).
File structure
• Top‑level files and directories:
• LICENSE, setup.py, README.md — project metadata and packaging
• .github/issue_template.md — contribution guidelines
• bert4keras/ — core library modules
• examples/ — ready‑to‑run scripts demonstrating usage
• pretraining/ — code to pretrain models from scratch
• Inside bert4keras/:
• backend.py — abstraction layer for Keras vs tf.keras
• layers.py — custom layer definitions (embeddings, attention, transformer blocks)
• models.py — model‑building functions (BERT, RoBERTa, ALBERT, GPT variants)
• optimizers.py — training utilities and custom optimizers
• snippets.py — utility functions including WebServing class
• tokenizers.py — vocabulary loading and tokenization
• The examples tree shows task directories, data files, and scripts that consume the core library APIs.
• The pretraining directory contains data utilities and a pretraining main script.
README for additional insights
• Highlights compatibility (Keras 2.2.4–2.3.1, tf.keras, TF 1.14+ / 2.x), supported pretrained weights, and installation/use instructions.
• References a web‑based documentation site under construction.
• Emphasizes frequent updates and extensibility for custom modifications.