How raywilliam46/finetune-distilbert works
Project type and purpose
• This is an ML-centric pipeline for fine-tuning a pretrained Transformer (DistilBERT) on a binary classification task (toxic vs. non-toxic comments).
• It uses TensorFlow 2.0, the Hugging Face Transformers library, Comet.ml for experiment tracking, and nlpaug for data augmentation.
Main architectural components
A. Data Ingestion & Storage
• Raw data folder (data/raw) with CSVs from the Jigsaw Toxic Comment dataset.
• Processed datasets (data/processed/unbalanced_dataset and data/processed/balanced_dataset).
B. Data Processing & Augmentation
• src/utils/data_utils.py implements loading, splitting, augmentation (via nlpaug BERT‐based synonym replacement).
• Jupyter notebooks (notebooks/gen_datasets.ipynb, grid_search.ipynb) orchestrate dataset creation and hyperparameter sweeps.
C. Model Training
• src/utils/train_utils.py defines the training loop on TF2.0, setups of DistilBERT + Dense layers, loss functions (binary_crossentropy or focal_loss).
• Two training entry points in notebooks: train_unbalanced.ipynb and train_balanced.ipynb.
D. Experiment Tracking & Hyperparameter Tuning
• Comet.ml integration in training utilities to log metrics (accuracy, AUC-ROC).
• grid_search.ipynb explores parameter combinations.
E. Model Artifacts
• Saved models in models/unbalanced_model/saved_model.pb and models/balanced_model/saved_model.pb.
• Figures in figures/ capturing results or learning curves.
F. External Dependencies & Libraries
• Hugging Face Transformers, TensorFlow 2.x, nlpaug, Comet.ml SDK.
Relationships and data flow
• Processed Datasets feed into Model Training (train_utils + notebooks).
• During training, training artifacts and logs flow to Comet.ml and Figures.
• Model Training writes final Saved Models to the models/ directory.
• Optionally, grid_search loop wraps around Model Training for hyperparameter sweeps.
Architectural patterns & design principles
• Pipeline pattern: clear separation of ingestion → preprocessing → training → evaluation.
• Modular code organization: src/utils for reusable functions, notebooks for orchestration, distinct folders for artifacts.
• Experiment reproducibility: Comet.ml for tracking, notebooks versioned via Git, requirements.txt for environment.
Essential elements
• Clear labels: “Raw CSV Data,” “Data Augmentation,” “Training Loop,” “DistilBERT + Dense Head,” “Comet.ml.”
• Blue for data,
• Green for processing components,
• Orange for training/model code,
• Purple for external services.