# raywilliam46/finetune-distilbert architecture

> Architecture diagram of the GitHub repository raywilliam46/finetune-distilbert, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raywilliam46/finetune-distilbert
- Repository: https://github.com/raywilliam46/finetune-distilbert

## Overview

##### Project type and purpose
- This is an ML-centric pipeline for fine-tuning a pretrained Transformer (DistilBERT) on a binary classification task (toxic vs. non-toxic comments).
- It uses TensorFlow 2.0, the Hugging Face Transformers library, Comet.ml for experiment tracking, and nlpaug for data augmentation.

##### Main architectural components
A. Data Ingestion & Storage
- Raw data folder (`data/raw`) with CSVs from the Jigsaw Toxic Comment dataset.
- Processed datasets (`data/processed/unbalanced_dataset` and `data/processed/balanced_dataset`).
B. Data Processing & Augmentation
- `src/utils/data_utils.py` implements loading, splitting, augmentation (via nlpaug BERT‐based synonym replacement).
- Jupyter notebooks (`notebooks/gen_datasets.ipynb`, `grid_search.ipynb`) orchestrate dataset creation and hyperparameter sweeps.
C. Model Training
- `src/utils/train_utils.py` defines the training loop on TF2.0, setups of DistilBERT + Dense layers, loss functions (binary_crossentropy or focal_loss).
- Two training entry points in notebooks: `train_unbalanced.ipynb` and `train_balanced.ipynb`.
D. Experiment Tracking & Hyperparameter Tuning
- Comet.ml integration in training utilities to log metrics (accuracy, AUC-ROC).
- `grid_search.ipynb` explores parameter combinations.
E. Model Artifacts
- Saved models in `models/unbalanced_model/saved_model.pb` and `models/balanced_model/saved_model.pb`.
- Figures in `figures/` capturing results or learning curves.
F. External Dependencies & Libraries
- Hugging Face Transformers, TensorFlow 2.x, nlpaug, Comet.ml SDK.

##### Relationships and data flow
- Processed Datasets feed into Model Training (train_utils + notebooks).
- During training, training artifacts and logs flow to Comet.ml and Figures.
- Model Training writes final Saved Models to the models/ directory.
- Optionally, grid_search loop wraps around Model Training for hyperparameter sweeps.

##### Architectural patterns & design principles
- Pipeline pattern: clear separation of ingestion → preprocessing → training → evaluation.
- Modular code organization: `src/utils` for reusable functions, notebooks for orchestration, distinct folders for artifacts.
- Experiment reproducibility: Comet.ml for tracking, notebooks versioned via Git, requirements.txt for environment.

##### Essential elements
- Clear labels: “Raw CSV Data,” “Data Augmentation,” “Training Loop,” “DistilBERT + Dense Head,” “Comet.ml.”
- Blue for data,
- Green for processing components,
- Orange for training/model code,
- Purple for external services.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Layer
    subgraph "Data Layer"
        RawData["Raw CSV Data"]:::data
        UnbalancedDS["Processed Unbalanced Dataset"]:::data
        BalancedDS["Processed Balanced Dataset"]:::data
    end

    %% Processing Layer
    subgraph "Processing Layer"
        GenDatasets["gen_datasets.ipynb"]:::processing
        DataUtils["data_utils.py"]:::processing
    end

    %% Training Layer
    subgraph "Training Layer"
        TrainUnbal["train_unbalanced.ipynb"]:::training
        TrainBal["train_balanced.ipynb"]:::training
        GridSearch["grid_search.ipynb"]:::training
        TrainUtils["train_utils.py"]:::training
        Comet["Comet.ml"]:::external
    end

    %% Artifact Layer
    subgraph "Artifact Layer"
        ModelUnbal["saved_model.pb (Unbalanced)"]:::data
        ModelBal["saved_model.pb (Balanced)"]:::data
        Figures["figures"]:::data
    end

    %% External Dependencies
    Requirements["requirements.txt"]:::external

    %% Data Flow
    RawData -->|loads| GenDatasets -->|uses| DataUtils
    DataUtils -->|produces| UnbalancedDS
    DataUtils -->|produces| BalancedDS

    %% Training Paths
    UnbalancedDS -->|input| TrainUnbal -->|calls| TrainUtils
    BalancedDS -->|input| TrainBal -->|calls| TrainUtils
    GridSearch -->|wraps| TrainUtils

    %% Artifacts & Tracking
    TrainUtils -->|saves model| ModelUnbal
    TrainUtils -->|saves model| ModelBal
    TrainUtils -->|generates| Figures
    TrainUtils -->|logs metrics| Comet

    %% Dependencies Links
    Requirements --> GenDatasets
    Requirements --> TrainUnbal
    Requirements --> TrainBal
    Requirements --> GridSearch
    Requirements --> DataUtils
    Requirements --> TrainUtils

    %% Click Events
    click RawData "https://github.com/raywilliam46/finetune-distilbert/tree/main/data/raw"
    click UnbalancedDS "https://github.com/raywilliam46/finetune-distilbert/tree/main/data/processed/unbalanced_dataset"
    click BalancedDS "https://github.com/raywilliam46/finetune-distilbert/tree/main/data/processed/balanced_dataset"
    click DataUtils "https://github.com/raywilliam46/finetune-distilbert/blob/main/src/utils/data_utils.py"
    click GenDatasets "https://github.com/raywilliam46/finetune-distilbert/blob/main/notebooks/gen_datasets.ipynb"
    click TrainUtils "https://github.com/raywilliam46/finetune-distilbert/blob/main/src/utils/train_utils.py"
    click TrainUnbal "https://github.com/raywilliam46/finetune-distilbert/blob/main/notebooks/train_unbalanced.ipynb"
    click TrainBal "https://github.com/raywilliam46/finetune-distilbert/blob/main/notebooks/train_balanced.ipynb"
    click GridSearch "https://github.com/raywilliam46/finetune-distilbert/blob/main/notebooks/grid_search.ipynb"
    click ModelUnbal "https://github.com/raywilliam46/finetune-distilbert/blob/main/models/unbalanced_model/saved_model.pb"
    click ModelBal "https://github.com/raywilliam46/finetune-distilbert/blob/main/models/balanced_model/saved_model.pb"
    click Figures "https://github.com/raywilliam46/finetune-distilbert/tree/main/figures"
    click Requirements "https://github.com/raywilliam46/finetune-distilbert/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#bbdefb,stroke:#0d47a1,stroke-width:1px
    classDef processing fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    classDef training fill:#ffe0b2,stroke:#ef6c00,stroke-width:1px
    classDef external fill:#e1bee7,stroke:#6a1b9a,stroke-width:1px
```
