How nmihtrug/distilser works
Project Type and Purpose
• This is a research/code project focused on Speech Emotion Recognition using a knowledge distillation approach. Its purpose is to reduce model size and improve inference time while retaining competitive performance.
• The README and file structure indicate that the project supports both teacher and student training setups. It is designed to experiment with different model configurations (e.g., various BERT-based and distilled configurations in the src/configs directory) and embed audio features using VGGish (see the src/torchvggish directory).
Analyzing the File Structure
• Top-Level Organization:
• Data folders: “IEMOCAP_preprocessed” (preprocessed dataset files) and “IEMOCAP_embeddings” for audio/text embeddings.
• “notebooks”: Contains Jupyter notebooks for exploratory analysis (e.g., plotting data, cosine similarity, gradual attention using grad_cam, etc.).
• “scripts”: Contains scripts for preprocessing (preprocess.py), training (train.py and train_distillation.py for teacher and student training respectively), evaluation (eval.py), and even checkpoints.
• “src”: This is the heart of the system; it contains modular code for configurations (configs), data handling (data/dataloader.py), model architectures (models directory with losses, modules, networks, optimizers), and custom implementations of VGGish (src/torchvggish). There are also utility submodules for both PyTorch and JAX-based training (in src/utils).
• Architectural Patterns and Responsibilities:
• The configs directory clearly defines multiple model variants (teacher: bert_vggish.py, and various student configs like nanobert_vggish.py) so that the training scripts can instantiate a corresponding model.
• The data layer (src/data/dataloader.py) is in charge of feeding preprocessed IEMOCAP data to the model.
• The model layer (src/models and src/torchvggish) encapsulates network definitions, feature extraction (audio via VGGish in particular), and loss functions.
• The training and evaluation orchestration is done via scripts and trainer utilities (both in src/trainer.py and the torch/jax submodules) which combine model configurations with their training procedures and optimization routines.
• Data Ingestion and Preprocessing:
• IEMOCAP data stored as “IEMOCAP_preprocessed”
• Preprocessing script (scripts/preprocess.py)
• Data Loader:
• Module in src/data/dataloader.py that feeds data to the training pipeline
• Feature Extraction:
• Audio features processed through the VGGish model (in src/torchvggish)
• Model Components:
• Teacher Model (defined with the bert_vggish config) and accompanying network in src/models
• Student Model(s) (e.g., nanobert_vggish, minibert_vggish, etc.) for the knowledge distillation setup
• Training Pipeline:
• Training scripts: train.py for the teacher and train_distillation.py for the student.
• Trainer components and optimizers within src/utils/torch (and similarly, jax variants if needed)
• Evaluation Module:
• Evaluation script (scripts/eval.py) to assess the model using checkpoints saved in “scripts/checkpoints_latest”
• Configuration Management:
• Config files in src/configs which set hyperparameters and model architecture details.
Relationships and Data Flow
• The Data Loader (src/data/dataloader.py) reads these preprocessed files to supply batches.
• The input data then splits into two streams:
• Audio processed via VGGish (src/torchvggish) for generating audio embeddings.
• (Optionally) Text embeddings processed by a BERT-based module (or its distilled smaller version) based on the chosen config.
• The Teacher Model processes the combined modalities first. In the knowledge distillation setup, its outputs and/or internal representations are used as targets.
• The Student Model (from another config file) is trained, using a dedicated training script (train_distillation.py) that leverages a combination of standard classification loss and distillation loss comparing its outputs against the teacher’s outputs.
• Training routines are orchestrated by the trainer utilities in src/utils (which separate PyTorch and JAX implementations).
• Model evaluation uses the eval.py script, loading checkpoints to compute metrics on the test dataset.
Relevant Architectural Patterns and Technologies
• Separation of concerns is clearly followed by splitting the system into data handling, model implementations, training, and configuration management.
• The design leverages modularity: Having multiple configuration files allows rapid experimentation with different architectures. The use of both PyTorch and JAX utilities (under src/utils) suggests potential model training approaches.
• Knowledge Distillation: A key architectural principle here, where outputs from a “teacher” are used to guide a “student” model training while reducing model complexity.
• Technologies include PyTorch (for neural network and training), with inference supported via VGGish (for audio) and BERT variants (for text).