# nmihtrug/distilser architecture

> Architecture diagram of the GitHub repository nmihtrug/distilser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nmihtrug/distilser
- Repository: https://github.com/nmihtrug/distilser

## Overview

##### Project Type and Purpose
- This is a research/code project focused on Speech Emotion Recognition using a knowledge distillation approach. Its purpose is to reduce model size and improve inference time while retaining competitive performance.
- The README and file structure indicate that the project supports both teacher and student training setups. It is designed to experiment with different model configurations (e.g., various BERT-based and distilled configurations in the src/configs directory) and embed audio features using VGGish (see the src/torchvggish directory).

##### Analyzing the File Structure
- Top-Level Organization:
- Data folders: “IEMOCAP_preprocessed” (preprocessed dataset files) and “IEMOCAP_embeddings” for audio/text embeddings.
- “notebooks”: Contains Jupyter notebooks for exploratory analysis (e.g., plotting data, cosine similarity, gradual attention using grad_cam, etc.).
- “scripts”: Contains scripts for preprocessing (preprocess.py), training (train.py and train_distillation.py for teacher and student training respectively), evaluation (eval.py), and even checkpoints.
- “src”: This is the heart of the system; it contains modular code for configurations (configs), data handling (data/dataloader.py), model architectures (models directory with losses, modules, networks, optimizers), and custom implementations of VGGish (src/torchvggish). There are also utility submodules for both PyTorch and JAX-based training (in src/utils).

- Architectural Patterns and Responsibilities:
- The configs directory clearly defines multiple model variants (teacher: bert_vggish.py, and various student configs like nanobert_vggish.py) so that the training scripts can instantiate a corresponding model.
- The data layer (src/data/dataloader.py) is in charge of feeding preprocessed IEMOCAP data to the model.
- The model layer (src/models and src/torchvggish) encapsulates network definitions, feature extraction (audio via VGGish in particular), and loss functions.
- The training and evaluation orchestration is done via scripts and trainer utilities (both in src/trainer.py and the torch/jax submodules) which combine model configurations with their training procedures and optimization routines.

- Data Ingestion and Preprocessing:
- IEMOCAP data stored as “IEMOCAP_preprocessed”
- Preprocessing script (scripts/preprocess.py)

- Data Loader:
- Module in src/data/dataloader.py that feeds data to the training pipeline

- Feature Extraction:
- Audio features processed through the VGGish model (in src/torchvggish)

- Model Components:
- Teacher Model (defined with the bert_vggish config) and accompanying network in src/models
- Student Model(s) (e.g., nanobert_vggish, minibert_vggish, etc.) for the knowledge distillation setup

- Training Pipeline:
- Training scripts: train.py for the teacher and train_distillation.py for the student.
- Trainer components and optimizers within src/utils/torch (and similarly, jax variants if needed)

- Evaluation Module:
- Evaluation script (scripts/eval.py) to assess the model using checkpoints saved in “scripts/checkpoints_latest”

- Configuration Management:
- Config files in src/configs which set hyperparameters and model architecture details.

##### Relationships and Data Flow
- The Data Loader (src/data/dataloader.py) reads these preprocessed files to supply batches.
- The input data then splits into two streams:
- Audio processed via VGGish (src/torchvggish) for generating audio embeddings.
- (Optionally) Text embeddings processed by a BERT-based module (or its distilled smaller version) based on the chosen config.
- The Teacher Model processes the combined modalities first. In the knowledge distillation setup, its outputs and/or internal representations are used as targets.
- The Student Model (from another config file) is trained, using a dedicated training script (train_distillation.py) that leverages a combination of standard classification loss and distillation loss comparing its outputs against the teacher’s outputs.
- Training routines are orchestrated by the trainer utilities in src/utils (which separate PyTorch and JAX implementations).
- Model evaluation uses the eval.py script, loading checkpoints to compute metrics on the test dataset.

##### Relevant Architectural Patterns and Technologies
- Separation of concerns is clearly followed by splitting the system into data handling, model implementations, training, and configuration management.
- The design leverages modularity: Having multiple configuration files allows rapid experimentation with different architectures. The use of both PyTorch and JAX utilities (under src/utils) suggests potential model training approaches.
- Knowledge Distillation: A key architectural principle here, where outputs from a “teacher” are used to guide a “student” model training while reducing model complexity.
- Technologies include PyTorch (for neural network and training), with inference supported via VGGish (for audio) and BERT variants (for text).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion and Preprocessing
    subgraph "Data Ingestion and Preprocessing"
        A["IEMOCAP_preprocessed"]:::dataComp
        B["Preprocess Script (scripts/preprocess.py)"]:::processingComp
        C["Data Loader (src/data/dataloader.py)"]:::processingComp
    end

    %% Feature Extraction
    subgraph "Feature Extraction"
        D["VGGish Feature Extraction (src/torchvggish/)"]:::processingComp
    end

    %% Model Components
    subgraph "Model Components"
        E["Teacher Model & Config (src/configs/bert_vggish.py)"]:::modelComp
        F["Student Models Configs"]:::modelComp
        F1["DistilBERT (src/configs/distilbert_vggish.py)"]:::modelComp
        F2["MicroBERT (src/configs/microbert_vggish.py)"]:::modelComp
        F3["MiniBERT (src/configs/minibert_vggish.py)"]:::modelComp
        F4["NanoBERT (src/configs/nanobert_vggish.py)"]:::modelComp
        F5["PicoBERT (src/configs/picobert_vggish.py)"]:::modelComp
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        G["Teacher Training (scripts/train.py)"]:::trainingComp
        H["Student Training (scripts/train_distillation.py)"]:::trainingComp
        I["Trainer Utilities (src/utils/torch/trainer.py)"]:::trainingComp
    end

    %% Evaluation Module
    subgraph "Evaluation Module"
        J["Evaluation Script (scripts/eval.py)"]:::evaluationComp
        K["Checkpoints (scripts/checkpoints_latest/)"]:::evaluationComp
    end

    %% Configuration Management
    subgraph "Configuration Management"
        L["Config Files (src/configs/)"]:::configComp
        M["Config Utilities (src/utils/configs.py)"]:::configComp
    end

    %% Data Flow Connections
    A --> B
    B --> C
    C --> D
    C --> E
    C --> F
    D --> E
    F --> F1
    F --> F2
    F --> F3
    F --> F4
    F --> F5
    E -->|"triggers"| G
    F -->|"selected config"| H
    I --> G
    I --> H
    E -->|"distills"| H
    G --> J
    H --> J
    J --> K

    %% Configuration Management Connections
    L --> E
    L --> F
    M --> L

    %% Styles
    classDef dataComp fill:#AED6F1,stroke:#000,stroke-width:1px;
    classDef processingComp fill:#A9DFBF,stroke:#000,stroke-width:1px;
    classDef modelComp fill:#F9E79F,stroke:#000,stroke-width:1px;
    classDef trainingComp fill:#F5B7B1,stroke:#000,stroke-width:1px;
    classDef evaluationComp fill:#D7BDE2,stroke:#000,stroke-width:1px;
    classDef configComp fill:#85C1E9,stroke:#000,stroke-width:1px;

    %% Click Events
    click A "https://github.com/nmihtrug/distilser/tree/main/IEMOCAP_preprocessed/"
    click B "https://github.com/nmihtrug/distilser/blob/main/scripts/preprocess.py"
    click C "https://github.com/nmihtrug/distilser/blob/main/src/data/dataloader.py"
    click D "https://github.com/nmihtrug/distilser/tree/main/src/torchvggish/"
    click E "https://github.com/nmihtrug/distilser/blob/main/src/configs/bert_vggish.py"
    click F1 "https://github.com/nmihtrug/distilser/blob/main/src/configs/distilbert_vggish.py"
    click F2 "https://github.com/nmihtrug/distilser/blob/main/src/configs/microbert_vggish.py"
    click F3 "https://github.com/nmihtrug/distilser/blob/main/src/configs/minibert_vggish.py"
    click F4 "https://github.com/nmihtrug/distilser/blob/main/src/configs/nanobert_vggish.py"
    click F5 "https://github.com/nmihtrug/distilser/blob/main/src/configs/picobert_vggish.py"
    click G "https://github.com/nmihtrug/distilser/blob/main/scripts/train.py"
    click H "https://github.com/nmihtrug/distilser/blob/main/scripts/train_distillation.py"
    click I "https://github.com/nmihtrug/distilser/blob/main/src/utils/torch/trainer.py"
    click J "https://github.com/nmihtrug/distilser/blob/main/scripts/eval.py"
    click K "https://github.com/nmihtrug/distilser/tree/main/scripts/checkpoints_latest/"
    click L "https://github.com/nmihtrug/distilser/tree/main/src/configs/"
    click M "https://github.com/nmihtrug/distilser/blob/main/src/utils/configs.py"
```
