# salt-nlp/efficient_unlearning architecture

> Architecture diagram of the GitHub repository salt-nlp/efficient_unlearning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salt-nlp/efficient_unlearning
- Repository: https://github.com/salt-nlp/efficient_unlearning

## Overview

##### Project Type and Purpose
- This is an academic/experimental code repository focused on efficient unlearning in large language models (LLMs). The approach is based on adding lightweight unlearning layers (using a selective teacher–student objective) and a fusion mechanism to merge different unlearning modules.
- The README clearly states that the repository is the official “code and data” release accompanying the EMNLP 2023 paper "Unlearn What You Want to Forget: Efficient Unlearning for LLMs". Thus, the goal is to implement an unlearning framework that quickly updates LLMs after data removal, without full retraining.

##### Analyzing the File Structure
- At the top level you have typical project files (LICENSE, README.md, .DS_Store) and a “src” directory.
- Inside src, the “models” directory includes both:
- A run.py file (entry point for execution) and corresponding Makefile and configuration files.
- A very rich “transformers” directory. This directory mirrors the structure of Huggingface’s Transformers library, with many submodules including commands, data processing (datasets, tokenizers, pipelines), multiple model implementations (e.g., bert, gpt2, t5, etc.), benchmarking tools, and even parameter-efficient finetuning modules.
- There are also branches for parameter-efficient-finetuning modules and custom testing, documentation, and utility scripts.
- From the structure, it is clear that the design is highly modular. Core functionality is built upon the transformers library, and then extra components (e.g., commands for training, unlearning, merging, benchmarking) extend that functionality.

##### Key Architectural Components to Capture

a. Entry and Orchestration Layer
- Main execution script (src/models/run.py). This script handles command-line flags such as “--unlearn” and “--merge” to drive different flows.
- Components that parse command-line arguments and initialize environment settings.

b. Unlearning Logic and Modules
- Core “Efficient Unlearning” module that implements the lightweight unlearning layers. Even if the code is embedded among many transformer submodules, emphasize that there’s logic to:
- Insert unlearning layers (teacher–student objectives).
- Handle data selection, i.e., “forgot_data” vs. “retained_data”.
- Execute selective model updates.
- Fusion mechanism used to merge different unlearning layers, as triggered with “--merge”.

c. Underlying Transformers Library
- Multiple model implementations (BERT, GPT, T5, etc.) supporting a wide range of LLM architectures.
- Data processing components (datasets, pipelines, tokenizers) essential for model training and evaluation.
- Command modules and utilities that allow conversion, benchmarking, and deployment (commands/, onnx, pipelines).
- Parameter-efficient finetuning support which is crucial for modifying only parts of the model.

d. Data and Configuration
- Data flow into the system:
- Input data (e.g., “forgot.csv” and “retained.csv” as provided to the run script).
- Downloaded data or pre-trained checkpoints (as noted in README instructions and via configuration utilities).
- Configuration and utility modules (files such as setup.py/config files) that facilitate environment management and dependency versions.

e. External Dependencies and Environment Layer
- External libraries and frameworks such as:
- PyTorch (and possibly TensorFlow components, given some TF modules),
- Huggingface Transformers,
- Datasets, spaCy, wandb for experiment tracking.

##### Relationships and Interactions
- That unlearning modules interact with the underlying Transformers library:
- They call model initialization, training routines, and data pipelines.
- They update model weights based on selective teacher–student objectives.
- From input data/distributed datasets into the model training/unlearning process.
- From model outputs to potential benchmarking and logging utilities.
- If applicable, include a feedback loop where benchmarking and testing modules inform parameter updates.

##### Architectural Patterns and Design Principles
- Modular Design: The project is organized into self-contained modules (commands, models, pipelines).
- Layered Architecture: Separate layers for data ingestion, core model operations (via transformers), and additional unlearning modifications.
- Extensibility: The extensive use of submodules (for many models and optional parameter-efficient finetuning) suggests a plug-in style where different models and commands can be interchanged.
- Abstraction: The repository leverages the abstract classes and interfaces from the Huggingface Transformers library to introduce unlearning functionality without re-writing the core LLM routines.

##### Final Elements
- Each component should have:
- A clear name (e.g., “Unlearning Module”, “Fusion Module”, “Model Conversion Commands”).
- A note on important technologies or patterns used (e.g., “teacher–student selective unlearning”, “parameter-efficient finetuning”).
- Be as detailed as possible: even though the repository is large, try to cluster closely related components (like all transformer model implementations) into one grouped module and separately represent command utilities.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Main Components
    Input_Data["Input Data (forgot.csv, retained.csv)"]:::external
    CLI_Entry["CLI Entry (run.py)"]:::orchestration
    Unlearning_Module["Unlearning & Fusion Module"]:::unlearning
    Transformers_Core["Transformers Core"]:::transformers
    Benchmarking["Benchmark & Logging Utilities"]:::external
    External["External Dependencies\n(PyTorch,HF Transformers,etc.)"]:::external

    %% Data & Configuration Subgraph
    subgraph "Data and Configuration"
        Setup_Py["Setup.py"]:::configuration
        Setup_Cfg["Setup.cfg"]:::configuration
    end

    %% Relationships
    Input_Data -->|"feeds"| CLI_Entry
    CLI_Entry -->|"triggers_unlearn/merge"| Unlearning_Module
    Unlearning_Module -->|"calls_models_training"| Transformers_Core
    Setup_Py --- CLI_Entry
    Setup_Cfg --- CLI_Entry
    Setup_Py --- Transformers_Core
    Setup_Cfg --- Transformers_Core
    Transformers_Core -->|"outputs_results"| Benchmarking
    External -->|"supports"| Transformers_Core
    External -->|"supports"| Unlearning_Module

    %% Click Events
    click CLI_Entry "https://github.com/salt-nlp/efficient_unlearning/blob/main/src/models/run.py"
    click Unlearning_Module "https://github.com/salt-nlp/efficient_unlearning/tree/main/src/models/parameter-efficient-finetuning"
    click Transformers_Core "https://github.com/salt-nlp/efficient_unlearning/tree/main/src/models/transformers"
    click Setup_Py "https://github.com/salt-nlp/efficient_unlearning/blob/main/src/models/setup.py"
    click Setup_Cfg "https://github.com/salt-nlp/efficient_unlearning/blob/main/src/models/setup.cfg"

    %% Styles
    classDef orchestration fill:#AED6F1,stroke:#1F618D,stroke-width:2px
    classDef unlearning fill:#F9E79F,stroke:#B7950B,stroke-width:2px
    classDef transformers fill:#D5F5E3,stroke:#1E8449,stroke-width:2px
    classDef configuration fill:#FADBD8,stroke:#CB4335,stroke-width:2px
    classDef external fill:#E8DAEF,stroke:#7D3C98,stroke-width:2px
```
