How feisun/bert4rec works
Main Components
• "Data Layer" – This includes the source files under the data directory (e.g., beauty.txt, ml-1m.txt, ml-20m.zip, steam.txt). These represent the raw user-item interactions or click logs.
• "Data Generation/Preprocessing Module" – Represented by gen_data.py and gen_data_fin.py. This module takes raw text data and generates the masked training input (TFRecords, versioned vocab files, and user history files) as seen in the README command examples.
• "Configuration Module" – The collection of JSON files in the bert_train directory (e.g., bert_config_ml-1m_64.json, others) which set hyper-parameter values such as hidden size, dropout, number of layers, etc.
• "Modeling Module" – The core of the project, represented by modeling.py, which likely contains the transformer network definitions, and optimization.py, which handles loss computation and training optimizations.
• "Training Pipeline" – run.py is the main driver for training the model using TensorFlow. This module loads the generated data, applies the configuration from the JSON files, and executes both training and evaluation steps.
• "Utility Modules" – Include util.py and vocab.py, which are responsible for various helper functions such as logging, processing vocabulary, and other common tasks.
• "Shell Scripts" – run_beauty.sh, run_ml-1m.sh, etc., which orchestrate the commands needed to generate data and kick off model training. These serve as entry points customized per dataset.
Relationships and Data Flow
• That the "Data Generation Module" produces several artifacts (training tfrecord files, test tfrecord files, vocab files, and history files) which feed into the "Training Pipeline."
• The "Training Pipeline" (run.py) should be depicted as central, loading both training data (from the data generation output) and hyper-parameters (from the "Configuration Module").
• Within the "Training Pipeline," illustrate an internal flow where the "Modeling Module" (model definitions from modeling.py) and "Optimization Module" (from optimization.py) operate together to train the BERT-based model.
• The "Utility Modules" can be shown as supportive blocks connected to both the data generation and training components.
• Optionally, include a note that the shell scripts serve as the start-up points, triggering the entire process.
Architectural Patterns and Technologies
• The architecture follows a modular design – each component encapsulates a distinct function (data preprocessing, model definition, optimization, etc.). Emphasize loose coupling where configuration and data generation are separate from training.
• Tools/Technologies include Python (2.7+), TensorFlow 1.12 (with GPU support), CUDA, and JSON for configuration.
• That this pipeline adopts a pre-training data generation step (masking for BERT) followed by transformer-based training, an approach common in modern deep learning applications.
• Annotations where necessary (for instance, include configuration details like hyper-parameter settings coming from the JSON files).